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The evaluation of the educational activities in which they are interested is a relatively novel idea foi!> 
national and internationa! development. assistance agencies. Barely ten years ago the major problem 
wa* to encourage the extension of the ^^arious components of educational sys^tems^ whose main defect, 
from the democratic, social and even economic point of view, ee<^med to, be thefr ijpi sufficient develop- 
ment. It is true that external aid .continues to be accorded today, as it was Vesterday, ta countries 
which ask for it and for objectives which they lay down with complete autonomy; but a dialogue has 
none the lesus been started between the receiving countries and the donor Agencies with a view to the 
better forrpulation of priorities in the choice .of projects to be financed or sectors to be developed - 
priorities y^hich are increasingly bound up with new educational policies* , 

In view of the volume of national and external financial resoi^rcee ifivolved, and their rela- 
tive, limits in relation to the needs,, this dialogue increasingly tends Rewards maximising the effects 
olsuch aid and of the educational policy in which it is integrated. With a view to the economic, social, 
democratic and other development of the cbuntr^j concerned; this search for effectiveness leads to 
the atteihpt to establish criteria, procedures and techniques desired to assess the value of the 
priority targets set, the adequacy of the means deployed to achieve them, and the qifality of, the 
results. * * - 

An ^evaluation component* therefor^ tends to.be introduced into all programmes and projects 
and, even- beyond that, into educatiqnal^^policies themselves. Is this really a novel feature? 

The idea of evaluation, however novel it may seen>, has always been present, under other 
names, in the oirganisation of educatfonal systems. Every, teacher and every institutionalised 
educational system h^s always sought to measure the results of the process of apprenticeship in 
which the pupils are engaged; d[aily marks, monthly essays, terminal examinations and even inspec- 
tion systems have always been^ among other things, checks on the 'internal efficiency* of educational 
systems. Similarly, through the formulation and progressive elaboration of curricula, educational 
administrators huve always sought, no doubt by an empirical approach, to meet the educational 
needs of man considered both as an individual and as a member of the contemporary social body - 
or of- a social group!" ^ * - 



Preface 



* Does the concept of evaluation, as currently applied tpday, differ fundamentally from the 
effort which all educational systems have always made, Ui more traditional forms, to assess, the 
value of their aims and their results? A too positiv,e''answer does not strike us as appropriate. It , 
'is, however, certain that the approach to these questions has been singularly modiried and enlarged, 
partly owing to progress in the techniques used and partly owing to a greater recognitioii. of the 
social ^d economic implicatio;is of the development of educational syscems. What used^o be merely 
a routine operation of supervision and progressive adaptation ot the system has become, in many 
^ respects, the medium for a re-jp»praisal of itsr aims and organisation. 

It would/ of course, be highly presumptuous to think that in doing this the second half of. the 
twentieth century is making a total innovation. Many countries in other ages, often on the occasibn 
of a political or social revolution, have had to face a renewal of their educational system as sweeps 
ing as any which can be envisaged today, but what is staking at present is the generalisation of this ' 
state of affairs both in the industrialised countries and in th,e developing countries. 

In the industrialised countries^ the progressive establishment of universal primary schooling, 
during the nineteenth century, ariS the subsequent development of secondary and higher education, 
responded to generous and liberating aspirations.as much as id economic needs, and the whole con- 
struction of the school system >yas Identified with democratic and social ideals. And now suddenly 
the school, which only yesterday was still regarded as the dispenser of all benefits and the object 
of universal respect, finds itself, like many other institutions, to some extent de^sanctified. The 
traditional attityde of asking "What is good for the schooj.?" i^ increasingly yielding place to another 
question, namely, "What is the good of the School?"" Naturally, this new attitude is bound up with 
political, social or economic factors specific to the different countries, ard the problem nowhere 
arises in quite the same term^. ^ • , * 

To som§ of the factors of challenge already known in the industrialised countries there are 
added, in the so. -called 'developing* countries, ^specific factors which are even hiore acute. 

Enthusiasm for the school as an instrument of liberation and human advancement, of economic 
and social progress, has not really waned, but we are faced wich a two-fold difficulty; in the first 
place, resources are lacking for the rapid construction of the desired school structure, and in the 
second place, existing educational systems are increasingly regarded, rightly or wrongly, as in- 
efficient and ill-adjusted to the solution of Social and economic problems as they arise in many" 
countries. Hence this search for new educational policies, new strategies bas'ed on a better choice 
of priorities, innovations in leaching, better use of existing facilities, and so forth* 

As has been said, a continuing process of assessment of the value of objectives, means and 
results is an essential condition of the rationalisation of educational systems in the developing 
countries. In theory this process should already be integratedTnto^he formulation ajid execiution of 
educational plans, where they exist; \n practice, even under the best conditions* the situation in this 
respect still remains very imperfect. V 

■ \ 5 ■ - ■ ■•• 

" ' . . -ovi ^' ' 



The Semiviar organised in\lay-June 1972 by the International Institute, with the support of 
SIDA. was not designed to offer partic^ants reacfy-made novel solutions affectipg the different levels.^ 
forms and stages of the evaluation process. More modestly, the Institute sought to conf.ne itself, 
on.the basis of certain concrete case stupes, to sensitising, if need be, the interest of the participants 
in the various possible or desirable aspects ^f evaluation and, above all, to encouraging a wide ex- ■ ^ 
" change of views on the im^plications of experience already acquired. The Seminar Report, excellently 
drafted by Mr. Jacqued Proust, sums up this week of discussions, during which it is hardly jiecessary 
to say that unanimity w"as not attained on aU points,^soyidely do the degrees of acceptance "of this 
concept of evaluation, still differ. \ . 

The general conclusion which emerges is neve^theles.s very clear; all the international or 
national aid agencies represented at the seminar and all the national experts invited in their personal 
capacity are conscious of the need to irr|pfove existing evaluation procedures, some of which hardly 
rise above a simple material verification of the exec"btion of projects. All the participants are j 
conscious, also, of the political and technical difficulties and \f the cost„of the effort which needs to J 
..be made. It is, of course, proper for each of them to develop il^ own action, but the general interest 
demands that the exchange of experience, launched particularly ori the occasion of this meeting, ^ 
should be actively pursfted'in future. It was suggested -that the IIEP'. in view of its interrtational 
position anjd its statutory function, should take the lead in this field. , , 

The International Institute WilLcertainly .act accordingly, withi^^ the limits of its means,* by 
developing its ow^i- experience in this field through the execution of its r^search programmes, and-by 
systematically continuing to organise meetings such as this seminar, duririg which {opics which Have 
so far only been touched upon can be successively studied in depth. ' , - 

Once again our thanks are due,to the SIDA which has provided us with the opportunity and 
the meaosjo organise this Seminar an^ which has made an extremely important jntellectual contri- 
■bution to the discussion; they are al80/due,to afl the other participants whose writ^n or 6ral 
contributions resulted in a frank, realistic and, in th6 last* analysis fruitful, exchange of views.. 



Raymond Poignant 
Director, IIEP 
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by J. Proust 



jHe Seminar organised with the support of the Swedish Int^niational Deve'lopinent Authority (SIDA) 

was held at the International Institute for Educational Planning. from Monday, 29 May. to JKriday, 

2 June 1972. ^ . ^ ^ « . 

The 14 aid.agencies (represented by 27, participants) and the 18 experts who attended the 
Seminar discussed and Exchanged views on the pijoblems raised by the assessment of aid for educa- 
tional projects, especially in. the developing countries. , *. 

^ The proceedings took the form of a round table discussion and the pre sient report is designed 

Jo summarize the debated, * • , . ^ 

'^^ In his introductory address, \he Chairma^n, Mr. Raymond Pqignant, Director of the IlEP, 
recalled the importance which should be attached to questions of assessment. He urged the need to 
extend the effort of assessment both to the education policies in which the dispersed projects or 
programmes financed by the agencies were integrated and J:o those projectii and programmes them- 
selves. He suggested that the first step should be to ascertain what improvements had been made in 
methods of assessing the internal effectiveness of education systems, which always largely condi- 
tit^ned the validity of assessment work. The Chairrrian desired to make it clear from the outset that, 
in assessing policies, there was no questioriof attempting to judge the educational as sistance policies 
of the different agencies, but, more generally, the aim was to assess the whole education policy in 

'which external multilateral of "bilateral aid was integrated. ' " ' 



The need for assessment 



At a time when "the world.education crisis' jy has been under discussion for some , years, it is 
ncgtural. to ask not only "what is good for the ^chool? ", t>ut .even more "what is the good of the 6chooP ' 
/even if the criticisms levelled against educational institutions are not univer;8ally valid. 
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\j Philip H. Coombs, 'The world' education/crisis; a systems analysis', 
.N6w YorJ^, London, Toronto Oxford University Press, 1968. 



3'. 



.8 



Evaluatiott of educatioml Bystems 



In the developing countries which are of more special interest to the aid agejncies repre- 
sented,. there certainly seenis to be flagrant maladjustment^ Among the arguments put forward, first 
place is given to the continual rise of unit costs and the inability to extend schooling, .From the point' 
of view of development, the question must be asked^ whether expenditure on education can go on^ 
increasing if it is desired to obtain the optimum allocation of resources. 

This state^of affairs means obtaining greater effectiveness at lower cost; that'is the whole 
problem of assessment. Unfortunately, this attitude fs not common In Ministries of Education. 

Furthermore,'' while it is agreed that; in general, education systems are often .top rigid and 
can survive only by innovating, it is quite clear that each innovation niust'be introduced only with 
caution and its effectiveness as well as its cost must be assessed as preci$ely as possible. The 
renovation pf education 'systenis wiU be feasiblf* and real only if appropriate assessment metjiocis 
are , finalised. ^ ^ " — — - — , ^ / 

There can be no improvisation ;in the matter of education and the many, experiments launched 
all over the world - such as educational television in the Ivory Coast should be assessed* A feed.- 
back between objectives and results, thanks to a process of assessment, would make it possible to 
.adjust, and, where necessary, to revise the objectives. , \ 

The need for assessnieht being recognised,, the question is,^ how to set aboutlt?^ * 

It was suggested that different levels or degrees^ qf assessment should be distinguished; , 
different forms and different moments in the process. 
. - ^ — The different levels _to_,b_€L.CQnsid(ered might be ; ♦ , 



- an overall level : this is the general diagnpjpis of the system, or, more 
generally, the level of analy,sis of education- policy. (5hly assessment at 
this level makes it possible to judge the choice of priorities and the ^neral 
results obtained; ^ s . 

- ah pperational level : this^is the assessment orprogrammes or projects. 

- The different form.q of assessment 
Throughout the seminar proceedings a slightly artificial, but useful distinction was drawn between 
^he internal e^fectiyeness ami the external effectiveness of education systems. This,.di«Unction i^ 
found in^ ihe assessment of internal effectiveness which measures performance in the light pf the 
system itg elf. Traditionaljly, success in final examinatiohs is Jhe priniarV elemfent ill internal 
assessment. The later proceedings of the sfeminar indicated that a number of o^ther elements are, 
or should be, now takeji, into accdunt in assessing internal effectiveness. 

In his opening address, the Chairman asked *what is the good of the school?' It does not 
seem possible to find !he answer ^without also assessing the external effectiveness of the system. 
This assessment should be made frpm a number of different economic, social £gid cultural points dt 
view. . . The aim should be to assess the quality of the 'output' of the education system in the light 
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• oV^yf'e ni^dM fifd hopfiU of society' and of indivi|iualg themselves/ Does the value iadded.to the indi- 
vidual' by the system meet the expedtations of the individual and of sogUty? Does it lead to progress, 
in the direction either of economic of cultural development? How can operation, criteria be applied * 
in such fields?. . . * \ . * . . 

These different levels and fonifis should be associated with dif fe rent . m om ent b at which the" 
assessment cm or should be made. In principle, assessment should be a continuous process, which, 
for a, simple project, may need several years. If this seems the ideal solution, it is perhaps not 
vejy realistic in the^case of modest projects, having regard to tlfe resources needed. Irwill be 
necessary to determine^the objectives of^the proposed, assessment, and very certainly to choose 
between them and to select tlie resources to be applied in the light of cost. / * ' , 
^ The nuniber andniagnitude of the outstanding problems ^re impressive'! * The Seminar did 

not'oprofess to solve them, but, more modestly, to compare the different points of view of Jhe a,ii 
agencies and the experts of s^me of the receiving countries on these. problems so as to claHfy them 
and bring'out the lines of action it is desirable and possible to follow in this field. 
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CHAPTER I 
THE ASSESSMENT! 



OF EDUCATION POLICIES 



If it is desired to assets the possible effectiveness of an aid programme, it is important to know 
whether it is well-timed. This timeliness depends on the context in which the programme,, 
(or project) is to be integrated and, in particular, on its adaptation to the education policy of the 
receiving country. This policy must therefore be analysed and assessed (or evaluated). It is hard 
to imagine an educational assistance programme, and still more a limited project, having a favour- 
able effect if it is not in line with the nation il education policy of the country assisted. 

Furthermore, the aid agencies all have official or unofficial aid policies and often preferential 
sectors for assistance* It is natural that they should prefer to act in keeping with their own policy 
and in thefr own sectors. | * 

For this purpose, they 'need not only a detailed analysis of the education system, but.also 
an assessment of education policies regaroed as an overall planned approach to^the development of 
the, whole education system. Such an asse^ssfnent should make it possible to judge all the conse- 
quences of the development of education systems taken as.^a whole on the different aspects of national 
• .life. ^ ^ ' ' 1 . • ' ' 

A number of papers in the seminar file 1/ proposed methods of sectoral analysis and the 
^Chairman stressed the^mpox^tance whic^ he felt sh6u]4 be attached to intra-sectoral and inter- 
sectoral coherence and he suggested Ipoi^ir-g.f'^T? opf:.'ational criteria to test the coherency of 
educa* ,n. policies. 



Discussion on this subject wari very outspoken and lively. 

* I i * " * 

While the need for an overall /assessment of educatior\, for any given country was not chal- 
lenged, the responsibility for making such an assessment was discussed. , The participants clearly 
indicated that the aid -receiving country*' had complete responsibility for such action> ,but could call 
upon optside assistance if it ffltlfie ileeJ From ~a^TecTmI«T~p^ ~~ ~' 



The representatives of the United Nations agencies particularly stressed that it was their 
, duty to encourage all useful st\idies jln this field so as to aid Member countries to make progress 
alpiig these lines.' 

On the other hand, it did Wot seem possible to agree on the lines to be followed by this 
assessnient. In the presentation^df this first head of discussion, assessment and coherence tests 
/were treated, in the sanie way. Some participants wanted to draw a. distinction on the ground that 
inter-sectoral and intra-sectoraL coherence could only indicate equilibrium^ but not the soundness of 
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that equilibriurr^l Siiverfel participants emphaiized thi^t the 'manpower approach* to educational 
planning had noj yielded the expected results, especially in the developing countkes. It would be 
dangerou^ to assess ah educational policy on the basis of t coherence test in this direction. 

Other , participants thoug^it it Wle to look for coherence between the spcial fiejd and the 
econai|iic field when the state of affairs in tiie developing countries seemct^l to indicate that this 
coherence was not to be found anywhere. 

Nearly all participants said that sector analysis should be very large and the ideal would be 
to incorporate the vrhole eocial sector and that, in default, in diagnosing education systems, it was 
desirable to assemble and analyse aU the data and studies available about .other social sectors. 

Sectoral analysis presupposes an analysis of existing objectives and Document No. 3 
suggests a method of approach to the analysis of aims, in six main phases 

- description of th* sector; 

- identification of the elements of policy; 

- distinction, specification (breakdown) of th.e elements of this policy (aims); 

r ranking of these. objectiveis or aimsj"^ 

i) - analysis of the links between different objectives at different -levels 

Of (tests-of means and ends); 

- analysis of possible conflicts or contradictions between objectives at the 
same level. 4, 

These stages should lead to the specification of the objectives-of education which^are often not veiy 
explicit in the developing countries. 

^ The paper presented by the World Banlv representative goes further and urges the need - if 
the aims and. objectives are too vaguely, expr/;8sed - to define alternative aims and objectives. This 
paper also raises the issue of the compatibiliiy^of the aims and,6bjectives pf education with the 
economic and social imperatives of satisfactory external productivity, thusVndicating that the 
Agencies are concerned by' the problem of coherence. 

This sti/dy of coherences, particularly inter-sectoral coherence, should, in the opinion of ' 
som e participa nts, make it po ssible to assess the effectiveness of a poli cy and,, In the opi nion of 
others, to judge the execution of a policy and therefore to assess the means of that policy. ' 

The question therefore is whether we are to judge a policy itself or merely its executioa 
without passingjmy value judgment on the- policy itself. ' ' * , 

As indicjated above, all the participants agreed on this second aspect of assessment which 
should make it possible to measure ^the gap which sometimes exists between the Plan and the facts, * 
Vhi'Jh leads to an examination of the objectives of education and, in the last analysis, to the question 
whethe.* the poli;oy followed is feasible. The disagreements referred to above may, perhaps, after « 
all be a false problem since it is quite natural to test the major disequilibria and effectiveness of a . 
policy bjr reference to ^its results. 
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i.. A. certi^n, number of fcHjleria.foi::. asseasing th<5 objectives of fui ed^ca^| on 'policy were s\;gi?, 

'getted in^iocument No. 1 presented by the |iEP. ^yithout underestimating the difficulty .qJ such an 

f iiercif^iVWc^^ in the light o/ the interdependence of the different facVors, 'miglijt lead to ^passing . 

'judgment qn the forms and methods of economic and social development chosen by the different 

coiinicieai, the idea of trying to assess the coherence of the objectiveB of education- in relation to Ut« ' 

: 'other economic and'isoicial.obje^tive^ of a given country is an approach which should be :ve?,y 

thoiroughly studied before being. dismissed ou^^ / \^ \\ \ . . 

^ ' While some participants voiced their apprehensions about a sonrietimes ambitioua global. , , 

ap^roadi and suggested. that rniero-ahalysis might yield good results, most of them .reco^ised that [ 

the micro and macro appr6ach were both indispensable and tha,t, in. particular, the glpbal study * . 

Should mrite it possible to assess economic coherence and to detect over-dange^pu8. d^8equi)ibrik 

which nif^t ariae'in the allpcation of resources and niight, in the medium term', comprotnise the 

external effectiveness ;of any education system « ' ' ■ ^ 

Some commentp were made on the<criteria suggested in Document No. 1. ItTb,. however,, 

worth citing some of them here which suggested , introducing^ indicators whicb do nojt seem, to have 

^ been used in the k|ipwn typologies. : * ^ / • ' ' f 

^in particular,' with regard to indicators of econpmic coherence, it was s^uggested thfit a n^w 

olassification of educational expenditure should be use3 which would bring put : ; . , 

- the/inert or cycUcal expenc[iture necessary, and sufficients 
? , ► the "systems, such as teachers/ pay and other runn&ig costs; , * 

^ ^ * ^ "^j- and, innovative expenditure or moderhisation or change. . ^ 

Another index a/ economic coherence is to^be^found in the. adjustment of the output of the education 

system to ihanpower heeds in conjunction with th(e pattern of incomes. / 

\^ 'With 'regard to the Elements of social: coherence, ^he members of the Working Party stressed! 

the need to work-out finer indexes,' especially for the regional'distribution of educational facilities 

and population. These-indexes should giye a clear picture of the territorial distribution of educational 

facilities in relation to population density. - ; . - * ' , 

The membef-s of the Working Party, vei^y conscious of the difficulties involved in the ... 

approach to the sociaLsebtqr, greatly.hope that operational. criteria will nevertheless be studied and 

that the more easily available criteria of economic coherence will not obscure this essential aspieci 

of the agsessment of ec^ucation policies. ' « y / : • , ' 

Aher discussing the elements of assessment of coherence within the different sectors. . 

(economic, spcial, cultural), the members of the Working Party would have Hked to tackle the , ^ . 

question of Unter-secftbral coherences, which would have meant studying the interrelations between 

the different criteria, but they thou^t that this came within the field of research and could not be 

i|i(idertakemin the context of the/present nrieetir.J. \* - 
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ASSESSMENT OF PpOGRijJRiMES AND PROJECTS 



CHAPTER 11 ' ^ 



A g^eat mddiy of the documents presented by participants in the Seminar related to the problems 
raised by the assessment 9f programmes ^d project^. It is quite certain that this is the field to 
which, the agencies are most sensitised .and where they have made the greatest methodological 
efforts l3ut there id. still a ^ee^t deal to be dons in many cases to get beyond the stage of a simple 
certificate of performance. For convenience of discussion^ it was agreed (as/stated at the beginning 
of the repprt) to. distinguish, no doubt somewhat artificially, between internal effectiveness and 
e;xternal effectiveness* * : ' ^' • <^ . 

The problems raised by the assessment of the internal effectiveness of education systems' 
were discussed at the Seminar for two days and<£^re the subject of Chapter III of this Report, 

. ^ The situation w|th rogard to the assessment .of extemapeffectiveness was summed up by 
the statement of the representative of an international agency to the effect that in the past his agency 
had taken account, of economic criteria only and was now trying to take into consideration certain 
cultural and socio-economic indicatprs, but thgit, in spite of everything, tjhie criterion of 'manpower 
production* was stiy tlie one'' which carried mqst weight in the final decision to finance a project. 

A certain nuinber of recent examples of .the application of the assessment process are given 
in the seminar documents and one of the most elaborate examples relating to Mie educational tele- 
vision project in .the Ivozy Coast was presented by/one of its principal authors. ^ 

The discussions ofi Documents Nos« 6 and 7, in parti<iular, gave a. broad .survey pf the 
metl^odplogies applied and the *compendiums' used. / ^ \ 

The thinking of the Seminar cenjtred mainly»on the^problems assessed tJie socio-economic 
impact of a,programme or proj^t pn a<given environinent. In this %vay, the question again arose pf 
defining the objectives of the\poiicy followed, since the results expected of a progr.amme cannot be 
judged crudely in cost-benefit terms, but rather more in the light of given policy. 

As noted in the previous Chapter, severaLai/encies recommend that special attention should 

be paid to studying the present aims and long-term objectives of the education system. If these 

aims and objectives are not sufficie/tly defined, then '*the difficXilt but necessary task must be under 

takeo of definirlg alternative aims Md objectives. It is only in the light of this new definition that 

< • . . \ • * 

the system can be usefully analysed'*. Some participants recalled what had been said about the 

responsibility for assessing education policies and stressed that the agencies should encourage , 

recipient countries to define their own objectives, according them technical assistance if necessary. 
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The sequences of the various assessment processes proposed in the seminar papers 
presented by the participants are substantially the same and take account of the "different levels, ' 
forms and moments'" distinguished by the Chairman in his opening address, l/ . 

The participants agreed that assessment should be a continuing process integrated in the 
education system. ItsShould be a permanentjeature of the planyiing process. They recognised, 
however; that under existing conditions/ neither the agencies.nor the national authorities could hope 
for such a result in the near future. * . , " <. 

They must therefore try to make some progress and move on frpni merely verifying the ' 
execution of a.prpgramme or piroject to^ a finer analysis of its effects. ^ 
The following stages were fo)md in the differen; alternatives proposed : " 
Preliminary sectot* analysis : 
, * Phase I : Stucfy of- aims and objectives. * ^ * 
Possible formulation pf alternative aims arid objectives. 
Pbase 2 : Structural analysis of ^education system. 

Adjustment of the nature of the system to the objectives defined 
.in Phase i, ' * , 

.Phase 3 : Analysis of the^:'technology' of education (teaching methods, 
management, staffing, decision-making, etc.), 

• Phase 4 : Attempt to organise and programme the.system so that it m^ts 
'the. objecti ves mpre satisfactorily, 

■* Phase 5 : Axialysis of costs and financing, and internal effectiveness. 

^ Phase 6 : PtSppsal of a programme of action iricluding a list of priority 

- projects and pre-inv^stment studies. 

During the project > . ^ 

In the caso of a large scale project- such tha{ presented at the seminar by the Ivory Coast, an ^ 
assessment systeni can, in some cases, ' be built into the project. It is a management tool which, by 
means of feedback, allows immedifirte corrections to be made throughout the project. This 'time and 
progress schedule' sys :m consj,st8 of collf|cting facts and figures and comparing them with the 
scheduled progress of the programme or project. But, as one participant pointed out, in order not 
to overload the system, it is important to know "what is worth noting and what is not". That is ^o 
doubt where the difficulties lie, sine* in ediacationaLmatters, the social' and economic consequences 
of a, project may becgmfe apparent only in the medium- or long-term and.it is difficult during the 
course of a project to judge what indicators will subsequently be valuable in assessing future changes 
in the behaviour pf a group, of individuals. 



1/ cf, page 4, paragraph 7, 
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Document No. '5 proposes a 'special assessment system |^consisting of verification in 
great6ir depth by means of micro-analytical case studies, and, in particular, the study and analysis 
of secondary facts. In this approach to the assessment of a programme, the important thing, ^n 
' the view of the authors of the paper^ is to rank in order the objects of the programme arid to examine 

& 0 " * . 

the interrelations bet ween^he different objects. Each object, constituting a stage in the edification- 
of^he project should be a stone i|i thai edifice and the^achievement of a number of objects should 
. lead |o the achievement of a. higher pbject. The chain formed by objects and means should lead to/ 
the main object of the programme.(see -Document No. 5). / 

The discussions on this field.of assessment were limited by shortage of tittie. It vet^ soon 
became apparent, however, that while it was relatively easy to measure the aspects relevant to 
intem&l effectiveness, the same was not true of external effectiveness, particularly if it was desired 
to go beyond the too narrow approach .of the equation *education-emplo;yment'. 

Research efforts should be undertaken along .these lines and the HEP is prepared to 
. co-operate actively in any action undertaken in this sector. 
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CHAPTER- III 



ASSESSING THE INTERNAL EFFECTIVENESS OF EDUCATION SYSTEMS 



This sector of the assessmerit'of education systems was introduced by'Yhree addresses u 

- Statistical inquiry into the internal effectiveness of education systems, by 
G. Carceles (Document No.. U). 

- Assessment^of education systems wit)i .a view to improving their internal 
/effectiveness, by N. Kravetz. 

- Examples'Of assessment; stu^ of the effectivjaness of the education system 
operations and development, by N. Davis. 

The first address, by Mr. G. Carcieles introduced the most^ recent work of the Unesco Office of 

Statistics on. the model« of internal effectiveness of education i^stems. 

This woxic, based on the measurement pf the ratio between the Intake and the output of 

education systems,, relates.to wastage. It studies the ratios between repeating and drop-out and the 

c 

age-sex ratio, etc. for each of these phenomena. This attempt to identify, the characteristics of 
wastage and its influence on real costs is particularly valuable .since it is dynamic and is concerned 
with flows rather than with statistics. Some points attracted the particular attention of the partici- 
pants,, especially '; , ^ * , 

- the prediction value of the rates of repeating^ and drop-out observed; ^ 

- failure or success in a given year of studies and its dependence on earlier 
J " results. . ^ 

The following is a summary of th6 main comrrlents made by the participants following Mr. Carceles' 
aiddress. * ^ , • ^ 

The different causes of the various kinds of repeating should be ascertained. Some 
instances of repeating and drop-out were due to-shojrtcomings in %e reception structure (schools noj^ 
providing ihe full 6ycle, fixed rates of transition,, etc. ). Some instances of repeating could be 
described as 'psycho]ogicar . Some teachers in fact thought that the subject they tau^t was difficult 
and the. time allowed was insvfficient. - * 

Some. participants stressed the pejorative- connotation of the terms employed, such as 
wastage and Hrbp-out. It would be better to talk of 'leaving*, ^which involved nq pejorativervalwe 
judgment. Leaving of various kinds might have very different causes, including economic, nutri- 
tional arid other causes which could not be attributed either to the school or to individual aptitudes. 
It was for tjiese reasons that the Unesco Office of Statistics spoke of ineffectiveness and not of 
wastage. ^ ^ ^ , ^ * t 

' 17', ^. ' . * 
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The addresses by Mr. N. Kravetz and MrSi N. Davis introdiTced the different tentative 
approaches now being studied, to the measurement: of the internal effectiveness of education systems. 

It was recalled in the first place, that it was essential, in order to undertake an assessnient 
with any hope of success, that the objectives of the education system to be assessed should present a 
certain numb^ of characteristics : they must be concrete, pertinent, feasible and quantifiable. 

Mr. N. Kravetz then introduced his indicators of the internal effe<;tiveness of education 
systems,'^ while makihg.it quite clear that while there was a wide range of factors which reflected the 
effectiveness pf educatioifi, those which were more specifically educational in character would provide 
the most valuable inforniation after they had been studied with care, and that it was therefore prefer- 
able to concentrate on the educational content jather than on the non-educative aspects. ^ 

The statement previously made by Mr. Carceles showed the value of statistical research 
into the flow of pupils. Mr. Kravetz proposed that^this work should be supplemented by research 
iiitb the causes of'repeating ani drop-out, a study of the possibility of.increasing enrolments at 
certain levels and of seeing whether the relations existing between these three factors could be 
improved^btherwise. than by making regulations. 

Jhe indicators to be taken into account if.it was desired to make tbis method not only 
diagnostic, but also therapeutic, must be regarded as the reflection of the decisive factors. The 
indicators of the i^jternal effectiveness of education systems were the result of conditions which 
existed within the system. 
^ . ^ The data on-enrolments and wastage were determined by other activities and by more funda- 

^^'mental relationships. \ ' 

The decisive factors in inte^rjial effectiveness were to be found in the following six categories : 

- pupils \ ^ 

* 

- teachers and other staff 
^ * V'. curriculum' - * 

- administration-^ ' * 

premffiies ' « ' 

- parents, -community and other clients ' 

These categories reflected Mr. Kravetz»s initial assumption that .*Vhat happens to the pupils is the 

essence of what happens to an education system.'* *^ 

The factors which determine internal effectiveness could thfen be considered within each 

category. * '^^^^ 

Thus, with regard to pupils, Mr. Kravetz'distinguished five decisive factors 

* * * 

* 1. Access t . 



• , ^ - Enough placeis ' , 

• . r Enrolments by sex 

- Enrolments by. rfegion* 

ERIC • 13 .. ' 18 
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Enrolment's by social status category . v. ^ 

" Existence of schools providing the full cycle ^ 
' , . - Objectivity of methods of selection and proniotion 

. , ' V 2. Assiduity ^ 
' ^ - Attendance of enrolled pupils ' • , ' 

- Compulsory schooling 

- Length of studies before leaving 
* - Re-enrolment of pupils who have left 

3. Pupils' results. * ' , 

- Determination of idegree of knowledge and progress mad.e 

- Determination of. degree and progress of affectivehess «^ 

- Determination of aptitudes 

4. Services 'rendered by the schooLto pupils ^ 

- guidance, pupil-teacher relations 

- boarding and lodging facilities 

- health services*-.- 

- transport ^ . 
• other services - . , 

5. Pupils' reactions . 

Pupils' statements of their own impression of the effectiyeness*^of the 
schood ' / 

- Behaviour and actions of pupils during schooling 

Mr. Kravetz theJi distinguished various factors which should be considered for each of the six 
decisive catfegorilBs he ideritified. He stressed the direct and indirect relations between the different 
elements which may have a decisive effect on intemfel effectiveness., ( 

The speaker emphasized then in addition to the six categories distinguished which determine 
and condition the effectiveness of eciuca£io;i, twjo supplementary factors must be.added^ namely 
educational i^esearch and development, as well as the planning of the system which,^ althought they 
are nat operations Jjirectly connected wijh teaching institutions, ijevertheless include functions whl.ch 
may have an infl\ieh<|e on the effectiveness of the system. 

The l.ast br^ch o'f this, introduction related to the measurement of the decisive indicators. 
Some of these criteri^ cbuld be n^esisured but others with a purely qualitative aspect were harder to. 
quantify. ' \ p • ^ „ > 

Mr,' Kravetz thcfti suggested a certain number of measurement concepts which' should, m^ke 
it possible to establish, relations between different elements. He proposed making use pf the whole 
arsenal of modern tiechWques, such as PERT, observation techniques, analysis of perception?, 
sociometric methods. \ /f O 



Report of the Seminar 



While it was recognised, *"^r, Kravelz said, that the application of aptitude tests and 
knowledge tests was now feasible (provided the necessary resources are available), .th^ measure- 
ment pf criteria of an affective character reflecting attitudes, interests, etc. was much more 
hazardous; it required th^ establishment of biographicaXrecords and scales of assessment and the 
availability of 'speci&lly trained staff, w*.. '.''^, ' ' * " 

In these circumstances, Mr. Krav'etz stressed that a great deal remained ,to be done to 
perfect measuring instruments for ihk educational process, but that, even at the, present stage, 
greater importance should be attached to the instruments already available, however imperfect they 

may be. * • - J. ' 

• * \ •* , " ^ ' • - ' < ' ' 

Mr. Kravetz ended his presentation by asking what role technical assistance' could play in 
^ '* • ' * ■ . ' . 

as8e8sing>the effectiveness of the education system. 

' . ' . ■ ^ • • 

In the discussion which follo)ved, a.nuno^ber of questions were raised. 

In the first place, the •volume of elements which niay play a.parjt in internal effectiveness is 
such that some participants asked whether the coist of such an undertaking would not be prohfbilive 
and whether one should not look for a few key -determinants to which preference should be given. 
Naturally, the whole range of .assessment techniques should not be brought into action until the : 
techniques with the highest priority and the greatest effectiveness have b|een applied. 

The leVel of a pupi\ is a highly relative concept and several participants recalled that the , 
objectives of education systems still too often. remained Elitist and in conflict with the present 
tendency towards mass education. Effectiveness meant attaining an jend which had been proclaimed, 
and it might well be asked whether the objectives of existing systenis were not outdated and too 
ambitious. This led to an apparent effectiveness which was often poor, 

The m^fiins of palliating this maladjustment of the objectives to the systeinl/ is perhaps, 
as suggested'by the Chairman, by Mr. Kravetz and by certain participants, to look for tl^e best ways 
of m^asunng the 'value added' to the child by education and in parallel to make the necessary effort 
lo lay ^down reSilistic and explicit objectives, , ' 

^ The last address on the assessment of internal effectiveness given by Mrs. N. Duvis, . 
indicated ^he bijoad lines of recent international studies in Xhi^ field : , . * 

- International educational achievement stucfy (lEA) - . 

. - National usses*ment of educational progress 
.. .. V * -VINIationaLachieyement testing programme • . 
-^International study of primary drop-out* 



.J.'/ this may, seem 'paradoxical sir)ce, in. principle, the system shoulcl adapt 
Itself '80 as '.to attain the* objectives.' * 
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These studies, especially lEA, have tried to nfeasure the. educational results achieved (knowledge o 
mathematics for the first phase of IEA) and to relate thjem to the.factors which seem to influence 
them : socio-economic environment, ;schQol, other environment, with a view to'trying to measurje 
the, productivity of the different teaching systems and methods^' ? 
• The teqhniques used are standardised tests and fchecks. 

Mrs. ^Davis then rjecalled that until they could generalise the standardised tests throughout 
'their territory,, the deveipping cciuntrries were constrained to use the results o| traditional examina 
tioris set by teachers who were mostly untrained an& inexperienced In the matter of examinations. 

Some attempts were being m'ade in the developing .countries to work out national tests 
designed to^ini'prove the assessmentsOf individuals in relation to their. aptitudes;. non-verbal tests of 
thejforniation of concepts, understanding te'sts, etc. In.general, there was a tendency to depart 
from th^ classical questipns of the dissertation type in. favour of simple.objective tests which can bi 
niarked mechanically; ' ... ^ . t 

V The speaker went on to say that the conclusions of the National Achievement Testing, I^ro- 
gramme in the United States were that tHe usual examinations showed : ' . , 

- how far a pupil was above or below the average marks of the group with 

• which he was compared;^ / ' . > * */ . . . 

^ - how far the average marks-of a class or school were above .ot below the ^ , 

* ' average of the-group with which they were coniparedi , . * - V 

The subsidiaiy conclusions" were also very important : ' < ' > • 

' . r ' . ' - . . "A * ^ ; ' - 

^ there wafe a very marked difference between examinations designed to. . 
make a selection or sorting oxit among pupils and tests designed to 
measure how-far the objectives of educat.ion were\achieyed; 

- while traditional examinations relating to knowledge were designed to 
measure individual differences and to proyide exact avjerage^marks for *. 

, , classes or schools^ assessment eatercises could provide information on 

* the overall progress^ of schobl children arid not merely.data on the 
\ knowledge' of the 'average child*. , , / v 

Mrs. Davis coricluded>y calling attention to the, fact that, while, assessment at rialionaMevel was 

verjr ofteVi difficult, if not impossible, because the objectives set for education systems Ave're too 

comprehensive apd defied all analysis in concrete terms, yery important assessments were,, on the 

contrax:y, perfectively, feasible , for smaller units such as provinces, towns or establishments and, 

could greatlyThelp to improve effectiveness. \ , : 

The following comments were riiade in ,the coulri^e of subsequent discussion ; 

- it should be noted that the assessment of education was, on the one h^d, 
general in respect of objectives and specific in respect pf results . 
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- Some participants wondere^d whether there was in fact any fundamental 

* difference between traditional examinatiorig.at^ national level, and ^ <. » 
. standardised tests oi: examinations. Waigin'Ot ^he difference rather one of 

form? It wa9 frue that the objectives of assessment pursued were dkferent 
in the cases pr'ej&ented, bu! were the techniques used totally, differ'ent? ' ' 
. - ' ^ ^ ♦ ' ■ ; . 

- Other participants expressed different .and even <xontrary opinions. In . 

^ , the'ir view, the tKadiUonal examination and the test were two. different 

instrumrents which some pepple ttgardfid arf complementary ^but the 
examination, while it was an obstacle which might be motivating, could 
* f alsb'be psychologically dangerous since it laid too much stress on failure. 

, The examination judgecl the pupil whereas the test judged the tnanner in 

, ' which the message-had been conveyea. ' . \ , 

• * .. ' ' ' " 8 

, r - Assessment, if it was aimed at a better Kno\vledge and^^ where appropriate, 

measurement of the value-added by education, needed measuring instru- 
ments, and Te$earch in this field^seems far from complete. 

To sum. up these discussions 'on the assessment of internal effectiveness, it seeryis that the studies 

so far made are still too much in the.domain of research and thatjthe techniques used, while they aire 

not challenged, necessitate substantial resources which cannot always be found. 

The determinants of internal effectiveness presented by Mr. K. Kravetz are not challenged, 
but if they are to be operational, it is desirable tcadentify those which carry the greatest weight in 
measuring performance, and tb.give them the preference in.studles^^ 

while the assessment of the terminal pattern of skills of the individual is important for t>e 
society into which he enters, his initial pattern should also be known so that the value added can be . 
n\ ensured by»di/ference^ . ' / ' 

Finally, and as in the t^o pther chapters of. this report,, the constant concern for a b.etter 
definicion of objectives should be noted, without which any assessment can only remain very , 
superficial* ^ . • ... . • • i.' * , . 

Conclusion . - < . . - t 

It emerged .f;:onci the whole pi the discussions that the subject discussed at, this Round Table requires 
still further systematic exploration. The divergences of view which sometimes became manifest are 
symptomatic of the weaknesses of the methods and techniques at present available. 

In fact, except for cjuieStions of the measurement of intjernal effectiveness for which th^_^ 
experimental stage has been substantially reached, the other aspects of. assessinent (external 
effectiveness, long-term impact of a project on its environment, etc. ). are still in the fieJLd of 
methodological research and the. few rare Experiments wh^ch have been made with cost -benefit 
analysis cahnot.be deemed to'ha^e yieldedf.p.ractical results of general application. • 

.It was confirmed in the course of this Seminar that an essential preliminary to any assess- 
ment is the need for a more realistic and- concre.te .definition of the objectives of education, without 
whi6h. assessment can only be a certificfi^tiB of performaticei • / , . 
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Another easential preliplnalry on which the participants wfere, however, less categorical', 
is^th^ need to set the sector artkly.^i.s of ediic^^^^^^ the yrtder context of the whole of the" social . . 
sectdr and to'thi^ow light on .tW i^rhole by .an .aesessment of tbe.education.poUcy of the country in 
question* " - '\ \- \ ' ' ^ \' . 



This Seminar can be slaid ^.o have had the merit of bringing to light existing. shortcomings ii) 
the atfseiBsment of fdycdtion and^he pfir3;)cipants rec,omtnended the encouragement of research in 
this field and the training of, fpeciaijsts. They suggested that the HEP should epv'isage d^vQting ' ^ 
pArt of its future, programme of activities, both in research and training, tp these /questions. 

Finally, some participants suggested the sjtandardisation of the Information requested. from 
assisted countries 'and the centralisation of these datadn a body, such as theilEP, 
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POSSIBLE £;RITERIAl!^OR EVALUATING EDUCATION POUCIES ■ 



by Raymond Poignant . ' , / 

with the* assistance o/ Jacques Hallak and Jacques Proust 



INTRODUCTION 



ic quantitative and qualitative development of educational systems comes about by implementing a 
/combination of programmes or projects*!/, financed mainly by national resources and supplemented 
by various forms of national or international, external aic). These programmes and projects involve 
the various levels or sectors, of general, vocational and technical education ojrganised for the younger 



generatiofi or for adultn^, within the school system or •out-of-schoolT 



Each of tliese programmes or projects is one part of a whol\ and this whole reflects, im- 



plicitly or explicitly, the choice of prioritica within the education system or in favour of the educa- 
tion system. Evaluation pf individual programtfies or*'projects.' is a process which is begi^ming to 
be given serious consideration by many national or international organisations, axid most of the 
papers presented at this .seminar deal w^ih this type of-problem. Hpwever, it^is^a question whether 
on this level of evaluation '('programmes* 'and 'projects*) and in close relationship with it, another 
type of appraisal should not be superimposed which would cover the 'education.policies', conceived 
as an over-air and planned approach to jdeveloping the educaticHt systend as a> whole . Indeed,' the 

evaluation pf 'programmes', and ^projects', however vital it may be, can only partially portray the 
- % * ' * ' * ^ ' ' i . * ^ . ' - ' ' • '\ . , ^ 

V value and effectiveness of education systems, and it would undoubtedly be of greater value to br^able 
. \ ' ' . ' ! I ' . " ' , ' \ 

to judge an 'education policy^ as a w^ole or, in other words, the consequences of the development of 

th^ education system taken as a whole on the various aspects of the life of the nation.. 



1/ These programmes ahd projects are relatively clearly explicit when 
financed by external aid;'' when financed by the national budget they are 
generally thuch rhore, implicit, sayex^ere the cpuntry concerned uses 
programme budgeting, ^^hich is the exception rather Uian the rule. 
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Such an endeavour would appear superfluous insofaf as education is confeidered as beneficial 

o 

P^r that it constitutes the 'priority of priorities' and that the sole problem is to increase it with^ 
1**1'' ' ^ ••»' 

out having to consider anV further the consequences and effects of its development. Today, .this 

ftttitude^s condemned by aU those who, rightly or wrongly, but nevertheless in^kn Increasing ixiajo- 

rity, claini that 'the vital Ijuestf on is not to determine what is good for the school, but what the 

schodl'is good for'. ^^Vithout considering the radic^ criticisms of those who attaclc the educational 

.irtntitutions, it]\ff obvious that on all sides, including the most official international and national.'' , 
rq^oVts, more and more serious Cx*iticism is aimed at educational systems themselves. . ^. 

' FirUt, we shall briefly examine the nature of the main criticisnis currently levelled at edu- 
oational systems and investigate their objective value.. Secondly, we shall examine whether it is 
possible t o, replace these criticisms with objective Criteria which would truly bring about an appraisal 

,of the value ahd effectiveness of education systems and hence of the 'education polices' which govern 
them. ^ ' * • • , • / 0 



PART I : THE MAIN CRITICISMS LEVELLED AGAINST 
• ' . * t ^ c EDUCATION SYSTEMS ' • . " • 

The reports from which the criticisms are drawn 1/ whicfi are summarised . below do- ridj pretend to 
make philosojJhical judgements as to the general orientations of educational policies,. bi|t inviiriably 
^ consiart of concrete statements qn-the educational policies Jn developing countries,, and.can be 
ijrouped together-uhder the following mijh headings:, ^ . * - 

(a) 1?hey are .poorly adapted to development. > * ' 

\ Thg great majority of developing countries are primarily agricultural and yet everytliing in 

the education system combines to turji away from rural activities those who enrol in it.* The school 
appears, to >e 'a foreign boiy in the environment and an uprooting factor'^. This criticism is based 
on the drift bf th^ younger rural generations towards the urban centres, and it does in fact seem 
that while the school not the only factor in this rural exodus, ^it iTa one; and not the least; , 

Ev<«i where this education appears to be properly adapted, J^e imbalance observed be^tween 
' tfie efforts taade for school enrolment fn tJie towns (admittedly often iJlnder the more effective p^e- 
sure of a rlrjlng proletariat); and the poor results achieved In the* coyntryplde, woul(}.7idt favour this , 
" latter environment. ^ 

Th;i Structures of thes.e-educatlon systems, which are pfteri the heritage of a colonial ,^.a«t, 
and their content, where not enough efforts hav* been made towards adaptation In subjects othir than 
, histoid or geography, prepare the student and adolescent ipuch more for a 'wKlte-collar job',, i. e. 
one In the administration or tertiary sector, than for Industry or, certainly, for agriculture. 



l/.These crltlclsxns.are ta-ten from a numbei* of official reports which It 



1^-* ^ would be ofno interest .to analyse separately. 

. EH|C r • 22 ^ 26 
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Thete •y«tem« attempt to inttill knowledge unrelated to the^^uirenients of econoraic 
gl-owth, whereas it would ht better, espe^eiaUv at primary educa^tion le^el« to cultivatf the natural 
qtMditiM of adaptability of the childiW and to n^ter attitudes necessaz^ for deveiopmjent. 

At secondary and technical education levels' in particular, those writing the reports point 
o^i that there is almost invarisibly a discrepancy betweer. .Gaining and job descriptions. It dees in, 
fact q)ptar that the ^order "of importance of values of the subjects taught is unrelated to the current 
nWds for divejopmeiit^. This ofttA leads to a proliferation of students in the' arts and, in a few rare, 
cases, to the risk .almost of a surplus of scientists ahd ^ general shortage of techniciluis ()r hi^^er 
J^schnikiaAS. Eiigi. .ers are trained without the supporting technicians, and professors of literature 
when thi^ is^^ need for professors of technology. At the s%me time, the teachers that are trained 
•at gr<»at cost are the unconscious cause of the perpetuation of the 8y«tem. • , ' " 

, At all^levels, «ihe Inapporopriateness of training with respect to the jobs available leads to 
.uncrmployinmi of ^graduates and a; *brain»dra^^ * ^ 

> (b) They .are poorly adapted.to the meaiis. available 

There are few developing, countries witi. an education '(Joctrine', and very often the final 
objectives of the education systems eniployed%re far from explicit. The education policy in these 
regions takes the form of a linear expansion of the. system left behind by the colonising power. This 
use of aii Elitist system for mass education h^fi resulted in a deplorable output (high drop-out rate) 
. and a financial burden put of all proportion to the.posiAbilities of tlie State.^ The increasing share 
in national i^espurces ailocated.to fiducatipn results in certain cases in an annuatiixcrease iri expenr 
diture ofihree times the rate of increase of the GNP., The conclusion is,that the 'strategy of linear 
expansion of education systems cannot In these circumstances go on indefinitely. * 
* (c)- A low output results from the enormous m^ans <^mployed . 

^ The very fast growth of school enrolments has not resulted in very considerable relative 
ou^uts. The stagnation of enrolment rates in the developing countries^ together with the increase 
in ^e number (in absolute terms) of illiterate diildren, are not encouraging factors for the future. 

/ In addition/ the continuous increase in unit costs has not led to any notable improvement 
in the efficiency of the educational. sy stents icpnstrint drop -but rate, ^only slight increase in success- 
fui.gi^aduates, etc.). • | 

(d) The systepnt pro&uces large numbers of poorly adfapted and frustrated people.' ^ 
In alnxost all developing countries, the selection methods within the school system result 

^ ' ' " ' • A ' ' ^ „ i 

in a small priviledged minority still monopolising access to.social and economic power. The great 

. . . . • . • ^ . , , ' ( ' ' / 

majority^ who had hoped that, as in the recent past, the school would open for them the dooris of an , 

' ' > 

*A Ministration* which is now saturated, have no alternative other than unemployment or a^rural 

existence. Unemploynient gind parasitic dependency or the^ family are often the solution chdsen be- 



causf the school^ has not developed in the child the necessai^ abilities to progress easily and 

6 » • 
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dynamltally in his own environment, The school and .its authoritarian traditions do not favour 
creativity. The longer dne stays in»schooli liie.mor.e one is cut off from'the\ealities and constralntii 
. of life.. While, the school gives a feeling of seputity., at the same time it, stifles .natural talent. 
. (C) Out-of-school education Is poorly developed. , 
. These criticisms as a whole -&re levelled against scholastic s yst ems; nonetheless, some 
reports deplore that all too often educational policies set ihtir priorities to the exclusive benefit of 
school estabUshmerits and to the detrlnient of out-pf-school educational systems for the younger and 
adult generations. 1/ / * ^ ^ ^ 

i PART 11 : CAN CRITERIA FOR EVALUATING EDUCATIONAL ; , 

POliCIES BEJDENTIFIEb? . • , 

The question raised in the yejry title of :i?art Twp showi that within the framework of this paper we , 
do not claim to do m6re than set out the problem's and Indicate the rqaln paths that may lead to eva- 
luatlng educational paUcie;/. . . 

A- Should one evalua>^e In terms of objectives or shduldvcme evaluate the objectives thenaselves? 

(a) One can eetabi\sh criteria for evaluating an edacaUonal policy by direct reference to the 
objectives - hunjan, pedago^tical, social, cultural, economic, political, etc. - that the educational 
system as a whole, or Its vaHous subsystems' considered a^arately, propose to attain. This is 
the mefiiod genei»al)y applied wh ^attempting .to evaluate programmes or project*. 

This concept of evaluation assumes that one can;. *\ 

(I) Clearly Identify the objectives of the system or of Its various subsystems, 
^ ° which VVe generedly Implicit or couched in very generaj ter^ 

^ (II) establish Indicators or Indices wherever pne nee,d8 to assess how the 

objectives laid, down are attained or are in process of* attainment. 

(b) Ho*.yever, tHis Inltl^ concept of evaluation appears too^limltftd and assessment of the 
value of an educational pqlicy must^ extend to the objectives themselves. ^ ^ 

This level of evaluation Is in fact certainly present at the start of the proci^ss of Implement- 
Ing 'programnies* and 'projects' during which the appraisal of ^he objectives to be attained;>and their 
orcler of p.riority governs ^e decision to be taken right from 'thfc outset of the operation, 2/ ' 

Can thijs^iAethod be used for judging the objectives .of an educational policy as a whole? 



:ERJC 



ly- ^o far, this limited development has protected out*of<-school educational 
'systems froni any. serious crltical.analysls.' The development of appraisal 
of 'pr^jects^in thl^ area wouWunquestlona^ily enable a mote objective. 
ju\3gement to.J^e made. ^ ^ ^ ^ ' ' 

^ 2/ Hctwever, when the programnrie .or project represwits purely and simply 
an extension of the existing system (secqndary schools, primary schools) 
or ^ven one element in a current reform ('middle* schools), iot example, 
the objectives of the operatlofi concerned blend Into those of the general . 
educational subsystem concerned, ' ^ ^ ^ \ ^ 

ZGi . . ' 
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The difficulty here would appear greater, since in thj? last resort, because tiie various 

factors are'interdependent, it is a question of pasjiing judgement on the forms and metho^'of eco- 

nomic and social development selected by the various countries. Eyen if the political difficulty of 

such an 'exercise can be solved, the technical difficulty non^cless appears formidable : to whai . v 

objective references can orfi turn in order to judge the ultimate ends of an educational policy? In 

the last resort one could atteiiipt to single out criteria of a philosophical nature, but in order to be 

•pragmatic we shall prevlsiQnally leave these out of the discussion, though" without .belittling the 

- importance of such criteria. . 

Is there any logical pattern in the development of educational system? , 

Ip the face of the difficulty in making abs^ute judgements of the ultimate ends attributed 

to educational systems, one might attempt to conceive the logic al>path that development of e<luca- 

^ tlonal systems should^follow in order to keep step 7.^th the successive stages of economic and. 

social development, estimating the latter, for instance, with reference ta the evolution of the struc- 

ture of the active population, the GNP per ciqpita, etc., i. e. with reference to the requirements and 

means of the country concerned / This would lead to a kind of dynamic typology of educational ays- , 

• ' ' • ' . ' / * ' 

tems in correlation with the different ITe^els of developnient, and this typology could be referred to 

in order tcjudge i^ether educational policies are^adequate. 

To explpre such an approaph, it' is interesting to compare how the development of educa- 
tion systems has corne about historically^ bpth in the countries that* are today industrialised and 
ln'th.e 'developing; countries. Were one to follow today the scholastic history of the industrialised 
countries since the start of the 19th century, one would And that most educational systems have 
fcUuwtsd comparable paths, taking into account on the one hand the requirements and possibilities 
resulting from economic development and on the other the evolution of social structures and the. 
concept of democracy.^ As the French, exairiple shows, the rapid spread of industrialisation that 
occurred in the second part of the 19th century enabled primary education to be generalised but did 

o . ' • ' * * , 

not intme^ately lead to a correspondlni^ development of extended studies (iecondary and higher), 

1 , ' . * • * • 

,and the 'school explosion 'jonly occurred much later, in particular following the second world war 

and doubtless. for reasons that were social rather than economic, • . 
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Table L France - Pattern of enix)lment rates by education level 
(1800^1968) and income per Capita. (US $) 



Level of^ education 


180a 


1900 


1945 


1960 


■ 1968 






' % 


' % 




% 


1st deirree v 
(6 - 11 years) 


30 - 40 


100 


100 


100 


100 


2nd degree 

(12- 17years)\ , 


1.5^ ; 


3 


12 


. 40 


73 


3rd de^rree < 
(18 - 23 yeax^s). ^ 


0,6 


1.2 


5 


7. 


13 



Per capita Income 1/ $ 135 



Per capita GNP 1/ 



450 $ 660 $ 1 383 $ 1 931 
856 1 806 . 2 541 



1/ Statistics previous to 1^0 irixould be considered as orders of 
magnitude, \ 



Table 2. Enrolment rates in C^lon, Tunisia, Tanzania and Iran and 
• GNP per capita ^ ' . ' 

■ ., ■ \ • ■; • ■ 

Level of 
^ educa-* 
tibn 



CEYrtidN TUNI'SIA^ TANZANIA 
1969 i969 1969 



- IRAN 
1960 1970 



% Age 
group 



% Age\ 
group 



% Age % Age ,% Age 
group group , group 



Primary 90.9 (6-13 ) 89 . 0 (6-12 ) 33.^ (? 13) 39,4 {6-11) 66.0 '(a-ll): 

Secondary V 

(1st cycle) 53:9 (1.4r^l5) 30.0 (13-15) 2;4 (14-17)% / 

(ll.O (12-17) 31.2^12-17) 

Secondary ' ' 

(2ndcycle) il,6"(16-17) 17.4 (16-18) l.ldd-lk)^ 

Higher * ,1.3 (18-21) 2,6 (19-22) 0, 3 (20-22)Yl, 8 (18-21) 3.2 (18-21) 



GNP per capita $ 165 



$ 229 



$ 95 A $ 213 



$,35b 



Table 2 gives a number .of similar statistics, though only for recent years, for xa group ot 
developing countries, * . \ . 

30 \ 



ERIC . 



26 



Possible cfiteria for evaluating education policies 



A*comparison of Tables 1 and 2 reveals that the growth of ^formal' education systems in 
the developing countries if occurring in a different way from the process that was followed in 
countries that are today industrialised. * 

The situation of enrolmait in France on the eve of the Industrial Revolution (1800) is fairly 
compax'able to that of Tanzania in 1969 and, whax is more, the level, of economic development of the o 
two countries is just about the same at these respective dates. 

However, this analogy breaks down at subsequent stages of development if we ref^r to^the 
examples of Ceylon, Tunisia and Iran; in these three countries, the development of secondaty and 
hi^ier education appears to have qccurred particularly fast. The development of education in Iran 
during the 1969-1970 period is particularly significant. With a level of economic development 
subtly below that of France in 1900 l/, Iranian enrolment rates fn secondary and higjier education 
in 1970 are respectively ten and three times greater than they were in France in 1900; in contrast, . 
primary enrolment in Iran in 1970 was far from complete (66 per cent). 

" It would seem t^at the 'social demand' for 8ec.>ndar^ aiid higher education amongst the new 
middle classes or the urban population has occurred in Ceylon, Tunisia and Iran.much niore quickly 
thafi in late 19th-century ^France. In other words, one might say that the factors of all kinds (social, 
economic, political, pedagogical, etc. ) explaining the sluggish expansion of secondary education in 
France in the 19th century are no longer present - at least to the same extent - in the more 
advanced developing countries . This situation is the outcome of a combination of new economic 
and social factors and probably at the same time of a different school policy . 

At tHe close of th^ I9th century, in most European countries, ^he primary sohccl was 
almost universal, while the secondary school was still restricted to the children of certain sniall 
social groups : at the time, this situation raised no serious political problems. 

Today, in the second half of the 20th century, the concepts of 'democratisation' and 
'equal opportunity' applied to hitler and secondary levels have lo varying extents provided the ins- 
piration for the education po^cies of the most advanced developing countries, following the example 
of the policies applied in the industrialised countries, and have led to rapid groMrth of second- and 
third- degree education, even though primary schooling is still not yet universal. 

To sum up, it is .evident that the various factors that in their time governed the slow pro- 
gress of secondary education in the 19th century, in the countries that are now industrialised, are 
not to be found an identical form in the 20th. century, and it is not to pe wondered, at that the 'deve- 
loping' countries today display a. relative advance, at comparable economic level, in secondary 
and higher education. However^ this observatiua throws little light en the logic of the policies, thus 
followed. - , « ' 



1/ To the extent that the 1900 French economy can be compared to the 1970 
Iranian economy. n , 
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,A« soon as one rejects the path followed by countries that are today industrialised a6 4 
frame of reference the only possible T^ray of assessing the value of 'the objectives of the educational 
systems in the devefoping countries is to attempt to judge their coherency with relation to other 
objective s and in particular those of an economic and social nature . In other/words. aside from 
any historical or geogi^aphlcal reiference, one should attempt to establish relationships of coherency 
between th^ development of education. and the economic, social and culturfQ development as a^i^oJ.e_^ 
for •ach coMntry, * ^< ' 

Can criteria be set for the coherency of the educational policy and economic and social 
development as a whole? . « \ 

In'practice, it is hard to see how the two approaches mentioned in paragraph A above 
(evaluation of results *rith relation to the objectives set and evaluation of -the objectives themselves) 
can be^systematically distinguished. Among other things, witH a view to establishing coherency 
between on the one hand the objectives of the educational system, the-means devoted to them, and 
the results gained ft^onti then^ and on the other hand the main economic and social objectives of a 
develqpmmt plan, it would not appear desirable to construct two separate systems of criteria. 

As a result, one can confine oneself to discovering.a set of criteria that would; embrace: ' 

^ - The naeahs devoted to euucation, assessed with reference to national 

resources, as a whoie; ' 

- the results obtained in the cultural, human, social, economic, etc. , 
fields. . . . . 

M er ely as an exarfiple, we list below various standard criteria th'at dould be retained. 

(a) Economic coherency criteria 

, ' J, , (i) Allocation ofrocourceBi^ftiiu-cbsts 

' - pj^pportion of national resources devoted to the educational system . • 

(in^GNP,;state. budget, etc.), \ . . 

shilre of educational expenclitures compared to other expenditures in the 
'social sector, ^> 

proportion of expenditures on education (considered as an investment In 
human^ capital) compared to investments in.pt^oductfve facilities, 

» breakdown of f ducational expenditures by level, 

- relat^ive shares of .expenditure on youth and adult ^education, 

- magnitude of unit costs (per pupil/y«»r l/) at the various levels in the 
educational system and factors which govern the magnitude of the costs. 

(ii) Coherency between the outputs of the educational system and manpower 
rekjuirements ^ # * ^ •! 

niagnitude oMnput and output flows of the system at the various levels 
and types of education,. ^ ^ 



,1/ Evaluted, for exaniple, in terms of GNP per capita. 
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- criteria concerning^'the integration into* the economy of those educated, 

- variotw, criteria concerning the quantitative and qualitative shortconainga 
of vocational and technical training compared to the requirements of tiie 
ecohoix\y, "etc. 

(b) Pedagogical criteria - 

• ratings 6f internal efficiency (see papers presented on this subject during 
the Seminar), * ' • . ' 

• ratings of t!|^uality of the teaching body, etc. 
• * . (c) Social coherency c'riteriV ^ 

- regional disparities in. the distribution of education, 

- social discrepencies and sexual discrimination, » 

- influence of\education on social mobility, 

- influence of education on the growth of the cities. 

(d) Criteria of a cultural nature 

r- the part playe^d by education in^cultural development, . 

* ? the 'adaptation of education, to the national culture (what one might call.the 
•isatioi}*- ofprpgramnies),.^ . . ' ./ 

- thi* proportion <}f foreign teachers in secondary and higher education, etc. 

(e) Criteria of a democratic and political nature . . • 

• ^ ' \ ^ .1 ' • • * ' 

, - the edu^cational system as a factor of national cohesion (the part played by 

' ^ education in citfzenffhip), 

^ - the educational system as a factor in participation by citizens in their . 

* ' responsibilities.' \ ^ • ^ " 

, • * * (f) Human, criteria . \ - ' 

\ . . i " - " * • " ^ 

It seems, in our view, that it is not ppssible to identify or isolate criteria for tiie part. 

* \ . ' • 

played by education in human fulfilment and happiness, even though this involves .neglect of a funda; 

mental asn'ecii of the consequences of an educational policy. ' 

* » ' "? ' 1, « . • 



^ . PART III r CONCLUSION - WHAT USE.COULD BE MADE OF SUCH 

CRITERIA? • ^ . , * ' , . 

The above enumeration is mainly confined to mentioning types of criteria; for eacli, more searching 
work should be carried out to specify its content' exactly. 

Right at the outset, it would appear that criticisms concerning the means employed and the 
results of an economic. nature apf)ear to^ be the easiest to establish (provided that the statistics f.re 
adequate) or, at least, those which can be quantified most easily; although delicate tp handle, cri- 
teria of a pedagogical nature also fall within an area that has already been partly explored. 
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In contrast, major difficultiei are. encountered in attempt&ijgf to establish indicators in the . 
politi6«i> social, cultural arid haman {|elds; they could be determined only as a result of surveys 
'cohducted.in the field by large teams of researchers. It is very often'a.question of analysing,the 
differences in behaviour of groups that have received some schooling and other. gr^ps irl order; to 
• judge the true effects of the educationalxsystem'T , . ; / 

Assuming that all the statistical and research tools hi^ve been successfully gathei^ed together 
to identify properly the- burden of the educational system on the ec«ncmy .and the ^rious types of 
result (good or bad) that society draws frorli it, to what type of conclusion, mi^t one tend? 

. It can clearly be seen where criteria of an economic nature.lead: , ' 

- hlghli^ting of the economic effect of schoolihg.and attempts to optimise' 

it, . ' ^ ' .'^ , ^ • ^ 

--highlighting of high, (or inadequate) unit do»ta# m - • 

- highlij^iinjg of the unsuita^ility of training imparted In relation, to actual 
requiremems,^ etc. ^ ' . • . 

However, the 'social' or 'democratic' criteria wUl highU^t disparities between region^^ . 
and social groups, the inadequate training for citizenship, etc; vid the conclusion that would per- 
haps have to be drawn from this type of criteria would be to develop education yet further and im- 
prove itrqualitv^ even though its economic burden, may already be excessive. 

Furthermore, analysis of tKe efforts made for youth and adults will generally ireveal the 
inadequate, provision for adults and thifi need* to do more. 

In fact, however, within the framework of a rhedlum-rterm' (5-year) plan, it is common 
knowledge that one canno^t do everything at ^h^ same time; all objectives cannot therefore be ^iven 
the same priority. In most casetf,. the objectives with the greatest economic return should have . 
priority J/ though without neglecting whajt can be done in favour of other types of objectives, within 
the limitations of the available means, the full use of whicli shojuld be aimed at, 

In other words, it is a question of attempting to identify major imbalances, the consequences 
* ^ ' ' ft""* * 

of which are particularly harmful to the general objeettves of development (too expensive an educa- 
tion system, output Oows poorly adapted to needs, destruction of national cultural values) in order to 
attempt to correct them. 

This approach does not imply that there is a sirigle*path for the development of education 
systerhs, even though in tjhe nature of things developing countries are faced .with fairly similar pro- 
blems. However, it is important that the path chosen is not in conflict with the other options chosen 
by each country within the framework of the over-all prospects for its development. 



1/ This realistic approach is the only one which in the longer term provides 
the possibility of successlully tackling more directly social, huma« and 
deinocratic objectives. * . « 
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TOWARDS THE DESIGN OP STRATEGIES FOR 1BVALUAT1NG 



\ ♦ ' DEVELOPMENT ASSISTANCE 

; ' '* •. ' byA.R. Mackinnom ^ ' ' 



Any Strategy devised for evaluating development assistance cfjri'ies with it sets pf jassumptions and, 

cto^tifaints and a nexus of objectives which can, range froni cpnci:ete production targets through to, 

abstract, metaphysical goals for the future of man. Expectations on many leyels exist in the minds 

of t^ose requesting development assistance in^educa&off - from th« learner who has s^ldorii any 

choice in w^at is happefning through to the governmental decision-maker who must :baian.ce set?, of 

alternatives of many unegqual weightsi. For thoke in charge of aid policies, expectations inay kaye 

only tangentiaj congruence with the .expectatic>nd embodied in .the request: general deveVp^*nt. , J 

policies are frequently outside Mil- co^ritrol^ he ifa suHject^ed to diffuse expectations from thje^arena^^lc,' 

public kccpljiitability; he is conUnuai][y harrassid by either a paucity or a plethora of irifprmatioa. 

which^may/pr m^y not bear oij his.responsiSilities. The difficulties are manifold and the pressures 

are gr«at fbi* doing almpst anything^is long as iomething is done. ^It is understandable, accordingly, 

why there has been so little satisfactory eva^uai;icn.!cr developnient assistance. 

The active verb ^evaluating.' has been ^lised.in the title of this discussion paper rather than 

the noun *evaluatioh' in ordelr to focus attention o^n an .active process: long-term evaluating which 

niight take account of what is a long-^angfs continually varying development process.' The extent of 

evaluation of any *time point cah.be expected to vary as well as the procedures of .evaluating. WJiat ' 

' • . . 1* * ' ' - f ' v» 

should be ainied at, accordingly, are evaluation, strategies which will produce evol^dng information 

systems relevant to the n^any 'facets of decision-making about development along a,i)rpadly-bas^d 

continuun^. depending on the decisions which piiiist be made, at any time point, the.cofiditions of 

\ ■ ! ' r ' ^ i '/ ' / • \ f '* ' ' ' ^' , , ' • . . ^ 
evaluation will be changed. If *developna<ent assistance* in a general sense involves qliange, tlT^en it 
\ " , *. '> • , . . ' • ' ^, / 

must be expected that objectives will shift wjlth the changing context. The evaluating strategists will 

^have to^va^y icbordihgly^ * ' ^ " ' . * ' ✓ . - , 

iai ^ - i ^ - \ ' ^ " . , ' 

Th^ste. general propositions aris^ from a growing. ^sensitivity to the pr'obtems of evaluating 
Canadian development assistance. Canada has had over twenty years* experience in the field of 
^educatiofi' assistance. Assessing ^assistance in terms *of the objectives which were set^at the outset 
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of the programme has been found to be .extremely cllifftcuU;. many devielbpment variables entered the 
" prdcesj; varied pjbogrammes we're occur;rilng:.simult^ recipient countri.es; changes in 

general aevelopiti^^ orientaiipn wefe extensiMe. **lijntilted aiiibitioni narrow perdeption 

• **rly years prompted! a Canadian policy of genf^iral offers.. When requests 

* f^''^l**''^'**«Pohsivenes's occurr^^^ As«umptions,.were made that the resources requested yftvt 
.fpprbfxriat* and wpuld.be available in the continijiing quantity and quality needed, U was also assumed 
that decisloA'^ made for sectoral alliocations would somehow;1)e'baIanced out in the recipient country. 
[ '^f^rf^^^i^^^^^^^^BJ^^Fitp/ Which have occurred oyer the past twenty years prompted several 

* 'policy •hlfts. '^Ge'M^ been replaced by an integrateci project approach. Areas of 

^pp^ciaHeaticiri.fpr^ti^^^^^^ been delineated. Above all else, a.broader view of 

Hey^lppment has'beguii to emerge, the reasons for such shifts are not hard to ilnd: unemployed 
' school leavers, for example, were occulting with klpirrning frequency; fragmented and disconjjtinuQus. 
assistance often meant that social and economic problems of a long-term hature..were 'left unresolved. 
In conjunction 1^ this awaren^sss^a re-thinking of evaluation strategies is now emerging. 

The approach is stili experimental iif nature and involves raising many basis questions. 
Can more. effective development assistance be attained thro^gh intensive stu<fy in the early planning 
stages? 'By paying extensive. and prolonged attention to ^prolalems enunciated by recipient countries 
and on a country- wide basis, would more appropriate information be generated for decision -making , 
within the country -concerned and for thoa^ in development ageiicies^ By building a. selective 
inventory of. emerging learning resources in Canada, could a more satisfactory base be set for 
conimuriication, for choices and for adaptation of^ the resources? 

Pppling of all available and current written information about a country from. as many 
iources as possible represents an initial procedural step. Although such information focusses 
primarily on development plans and problems, , historical, ^political and cultural information is also, 
included in til e review. The air^ to si&nsitize thp«e engaged in the evaluating process. Information 
not viewed as management daU which^might assure increased operational efficiency; the objective, 
brthe* colleption is'to 4n-form* evaluating teams^so that divergent information when it recurs can be 
perc(Ejt.ved ajB part of a board .dlc^elopment picture. Further, the serjcitizing is aimed at bringing 
together factors ^^^^ p€|.rhaps should never have been separated in "the. first . place. For example,, a 
narrow "sefctorfid approach to educa.iion can ifrequently cause fragmentatlpn and effprts are needed to 
Bee many aspects of human learning in many diverse settings ^s PW^. of a total development picture. 

Such inforniation m6s,t be supplern.ented soon by field site^visitst A teani approaclvhas been 
adopted as a basis for compajping percepfiohs, and for intrbducing ^various xiiscipUnes into the 
generation of information. There ai^e frequent additions made to the team depending on the current 
feature of the overall review. It r^xust be emphjEkaized that proj'ect definition is not_a feature of field 
site visits. Whether the visits centre on identifying educational Innovations in Canada^ or in 
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recognizing man|Jower^ development opportunities in a northern Nigerian . state, the^Watipnship, of. the 
problenfis identified to a. nattonal .^r i;egioni4 perspective it continuously sought. Above all^ qpntacts 

' are . spujSht^oult whip will ch ' . ' ' 

■' "■■"'*•». , . ' - . . , ' .>»**« 

After the field site visits* attempts ar.e made to put ^e information inta an acces alible, iovt^ . 
* ♦ ' ' ■ « ^ ' ' ' ' ' * . * "* * , ' . ' 

to suit many purposes. Decisions afe required, for example, at many administrative levels at home" 

. ^ ^ \ ' ^' . ' ' i I ' f / ; 

and abroad; there are short-term urgencifjs to bis met and pi^evious commitments to .be honoured. . 
The piHncipal task.is to reconcile the immediate issues with long-range planning. / Above all, active • 
commanicatipn muttt.be maihtaLpied betweiui those who are charged with the dey,el(»ment. policies in 
the countries concerned and those responsible fpr .i^d policies, and their, administration. ^ 

At this time point, shprt, intensive, ip-depth studies may be required as part of project 
definition. Since the investigation may be narrowed.down to a sp^illc sector such as education, . ^ 
the problem, is to maintain a balanced view, ot development in the.face of decisiom to be. made Jin an« 
isolated si^ctor. The luxury of in-depth studies cannot, bfe afforded in all the areaii Of development, 
wHich might impinge on the decisions. Itj is. imperative,. accor<^ngly, that decisions made for a 
given sector should include, evaluation procedures which vrill' include other important .developrpent. 
variables. Thus, when decisions are made for a t)roject in e^ca^ipn assistance,, evaluation must 
include effects on the spc^'al, economic and cultural life of the. country. The preliminary, intensive 
country study can often serve as the basis for setting out the appropriate variables and the 
procedures for evaluiktion.' • ' . .V/. ^ • 

Given the Ipng-term process prientation which was enunciated earlier,, there would.appear 

to bz justification for continuing tp explore such strategies oa an experimental basis. Our experience 

to date sliows that although the, context of studies will vary greatly from C9untry tp country, the . 

. . techniques employed in evaluating hold the possibility of larger applicability. It is possible th^t once 

techniques have been clarified,: much of the responsibility of evaluation, can be passed, over to. others, 

*y, * * ' *' • ' . ' • ^ . ' , . ' . ,ij . 

, particularly to thps.e in the countries concerned. The role of the evaluator then becomes one of 
, * ■ -^^ ♦ * \ • ' ' : ' ^ '■ ' 

refining appropriate pjrpce<jures of studies, adapting those whiph have proven to be successful^ and 

coordinating* the evaluating; prpcess as it moves through its many phi^ses on a broad front' of develop- 
ment assistance; , " ' .... ♦ * ; . ' ■ 

Three principal considerations 8e.emto hold some good possibilities for emerging strategies, 
pf evaluation: " , . • ^ . ' ♦ ^ ' 

li, .Each programme of the future which is aimed at 4^velppment assistance should have an . , 

, . ^ evaluation desi^ built in at the conceptual phase of pyroject proposal. This design will be, 
comprehensive and will provide for: assessment pfthe impact p^. the programme; modifi- 
cation of oa*going projects; and decisions concerning the future of a given programme 
* " (focus, scope, continuance, termination). * , ' * ' 
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2y ' Each evaluation design should be based on an analysis of the overall situational constraints 
".and barpiers ta programme effectiveness. This analysis should include .cpnsideration of 
conditions and resources in both the donor and the recipient countries. 
3. Future evaluatiO/i designs should pr.ovide for active pariicipation of the.recipien^t countries 

^ in the evaluation process, . • ' , * . 

We ire now at the stage of^experimentlhg with sucfi considerations, It is our hope that 
these general evaluating strategies will increase the quality of .deveJLopment assistance programmes 
by providing for an ihcreased flow of meaningful programme information between the recipiwt 
countries and the development agency. ' * . \ - 
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A SYSTEMATIC APPROACH TO GOAL ANALrSIJS AND EVALUATION 
OF SOCIO-ECONOMIC EFFECTS// WITH SPECUL REFERENCE TO 
EDUCATIONAL PROGRAMMES) ^ . ^ 



by G. Dahlgren * 



The main purpose o'f this paper is to discuss various methods to systeh.'^tize goal formulations and 
. ex-poi5t evaluation procedures within the public sector. . * ^ 

' Special attention will be paid to the methodological problems encountered when evaluating 
public inve&tmepts guided by social rather than purely*economic ^oals such as educational projects 
and programmes. , ' ^ • 

The present lack of social allocation theories, and evaluation methods is striking in parti- 
cular when considering that tht corresponding fields in„ economic theories constitute major research 
areas. The increasing importance -^f the public sector and a unified social and economic fipproach 
. to dev6lopmj5nt underline the urgent need for a systematic analysis in this field. 

Our level of ambitiofi as regards thesejmprovements mxxst however ue rather low as the 
theoretical base ic missing as well as our knowledge about many "^of the important, functional 
. relationships within tl:^ public sectpij. We simply do not know the linkage betw/eenV for example^, 
economic means and certain social entjs. . * , * , 

^ The first question to be posed is of course what methodological approach we^ car\ apply in 
order to systematize artd structure the problems to be discussed. ^logt of the available aiiw]^'ticai 
tools as regards evaluation have ^een developed within the private sector and are - explicitly or 
implicitly - usually based on profit oT growth goals. If these yaiue premises of the mefhod as such, 
are not recognised when sdcha method is transferred into tjje public sector, it may very well be 
that the method chosen rather than the public goals stated^ directs the allocation decisiopsj^and indi- 

^x)^t6s what should be considered a success or failure. <^ « ^ 

' ' , » 1. • ** * 

Assume^for example that we applied the't;:aditional niarket -based internal rate of return 
method as an allocative tool for educational programmes. ^ThiS methof^ presupposes that the bene* 
fits from the programme analysed are to be^-expressdd in monetary^ terms. The incomes created 
throu^ education thus becomjsi the benefit^s.to be considered. • . 



EvaQuation of educational systems 



I 



^ \ jProm this follows that the more income gener&ted through an educational programme the . 
. higher priorfty it should be given because a high internal, rate of return in economic growth terms 
is consider€f4:to be the best indicator of a.sound and productive investment. 

Let us now apply this philosophy to reaUty as .it exists. Is it in Une with the existing 
IM)Utical will to consider education of women less important than education of men in all societies 
where the men traditionally get the better-paid joLs^ 

Should irf-service training of hi^i income ni^it-club managers be given a higher priority 
in the development plan than up-grading of low inconie agricultural extension workers? ' • 

Only if the policy rmakers answer questions ^such as thiese with "yes, 6t course'' can it be 
assumed that their goals and^the iinplici't goals of the analyticaLtool kre in Une, with each other, v 
If they, cn the other hand, do''not agree to the stated priorities t^eait must also be recogiiized'that 
tKe market-based internal rate of retuxrn figure is h-^l a relevant indicator for establishing educa- 
tional priorities. anfl evaluating the success of a training programme. 

To claim - as sometim^^. happens - that the economies of the market are politically^ neutral 
only reveals either a narrow-minded ighorance as regards a'lternatives or more likely a 'technical* 
approach to policy-making. ' . ^ " ^ 

Instead' the evaluator mixst <Jerij/e^he benefits as well as costs from the poHtlcal goals 
stated. In order to carry out this task^it may be useful to systematize the various steps and. levels' 
of evaluation. o , • 

1 • A sysleinatic approach fo" eva?uation " \ */ ^ \ 

Evaluation is. a continuous process from the moment when^a project iclea-is barn to the day the 
r.esylts and effects of a certain project cannot be i-elated to the objectives of the project. Project; 
evaluation must thus be a continuous process integrated in the planning, implementation and follow- 
•jp phdses of the project."^ Thin siystematic approach to evaluation can be illustrated in the figure ♦ 
oix the following. page. , " » . o . 

In this figure, the left wing of the 'V; illustrates the planning phase starting ^ith the 
analysis of main objectives which are then concretized into* sub- goals, production targets, activi-" 
ties^ cost estimates and implementation plans. The wheel at the bottom of the feeds planned 

, * ' ^j"* " OSS' 

in-put into the implementation process. The right wing of the shows the follow-up of actual 
results and effects. Tl^e comparisons between planned. and actual resiults (l^vel 7 -3) is then 
defined as built-in evaluation and the comparisons between planned, and actual effects (level 3-11 
defined as special evaluatidn. . * * 
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PLANNING 

fMain objectives 

Sub-goals (incl'. es- 
timates of total costs) 

Product units 

6 

Production targets 
Activities 
Dfrect costs 



Plan of operatioi;i:Uncl 
evaluation planK ^ 




IMPL 



TION 



FOLLOW-UP 
Significance 

* Productivity 
Actual 'production 
Acti;sdiies carried out 

dual direct costs 

/ 

Actual in-put 



The V-model also enables us to distinguish between the fallowing four basic concepts which . 
are to constitute the cornerstones in the evaluation: 

' • . Planned significance^ Planned fulfilment of main goals 9 

^ ^ " '^Estimated total costs 

. ' . , - Planned productivity = Planned'production targets 

Estimated direct costs 



Real productivity 



- Real significance 



Real production 
Real direct costs 

Redgj fulfilment df main/ sub- goals <f 
R^l total costfi 



The terms will be used both in an ex-ante and ex-post meaning. To distinguish between the . 
two we. are going to talk abbut potential significance when applied to an ex-ante situation with poten- 
? tial goal fuyilment and potential (estimated) *otal costs. Rfeal significance is then the term used for 
ex-post analysis as it is based on real goal fulfilment and real totals costs. The same distinctioa is 
of course also made between potential productivity and real. productivity > 
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The present trend in most economic literature^ only jto consider th^ ex-ante concepts, is 
• , * * •» 

misleadirfg when not made explicit and can easily breace the impression that the difference between, 

for example, potential and reafsignificance is marginal and fhat,no ex-g^ost analysis Vherefore^is 
necfessary. Experiences, whenever observed and analyzed in a systematic manner,, almost with no 
exceptions sho>v how important it is to distinguish between data as estimated in the plan and as 
recorded in reality. • *^ ' ' • 

The Jfouftdatioh' of this evaluative system thus consists of the analysis of objectives, goals 
ai)d targets during the planning phase., ^ 

' . ^ . ' . 2, Analysis of public goals ^ ^ * , 

The importance of well-defined goals can bfe described in terms of: 

AUocative power ; ^ the extent to which criteria for establishing priorities 
^ ; \can . be derived, from the stated goals. . , . " 

Management power : the extent \o which programme activities c§n be 
retlSted'to and guided by the stated goalg. 

Evaluation pdwer : the extent to- which progress niade can be expi*essed 
' ^ ^ ^ » * in terms of the -skgtjgd goal$. ' 

The firsV^ep.in ah analysis of ah educational programme is thus to define as clearly as possible the 
^ policies wlfich are to form the base fbr'the allocation decisions, management control and evaluation 
work, • ' J . '* . • * ' \ ^ * 

^ Most analysts would theoretically agre> tb the statement that it is prerequisite for any 
analysis in the public :sector to know what the policy -mafcer^ want tfifc achieve - that ib beir^ familiar 
wit\i their goals*. But in practice few analysts - jin pary'eulat^hose .using Sfconomic research 
methods - start off their worjc by frying, to find out what these go^ls really are, 

^ This theoretical accefptahce but pra'cticarneglect of the political policies can partly be 
explained by the' fact that goals are Already implicitly g^yen in most economic models partly by 
difficulties to transform public polipy statements into operational goals. 

The above described evaluative approach by definition is not linked to any particular eco- 
nomic or social policy but based the stated policies in a particular country for a particular , 
sector or programme at a particular time, a'^goal analysis must always be carried out. In faijt, 
<> it constitutes the v4ty heart of the ?b^stfe'm. ' 

The analysis of public goals - as seen froni the cvaluators point of view - can be divided ■ 
itiio the following six major phases: * ^ " 

> (a) Describe the prbblem area \ 

' \ (b) Identify policy outputs ^ ' . 

. (c) Specify the goal elements 

<d) Structure the goal eleiV\ents . *• 

(e)^ Analyse the linkage goals on different levels (m!6ans and ends test) 
, , *(f) Analyse conflicts between goals on the feame leveL * 
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These six steps of the goal analysis, on which the following discussion will be based, ian 
be illustrated as below: , a ' * . 



A. Describe the pi^oblem area 



B., Identify policy outputs 



C. Specify, the goal elements 



D. Strtix^ture the goal* elements 



Analyse the linkages between 
goals on different levels 



F. Analyse conflicts between goals 
on the same level 
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2. 1 Description of the problem area ' 

Policies are not formulated in a vacuum. Th^ are - pr should at least be - based on an assessment 
of the present situation. The achievements to be aimed at. can then be indicated and transformed 
via, e, g; a political process into policy outputs; annual and long-term plans. An evaluator of, 

e. g. Educational programmes should not directly be involved in policy-making. His role is to uti- 
lize the policies stated in the analytical work in order to help, the policy -makers to acquir^e a better, 
understanding of alternatives as judged against their own policy statement. The evaluator can, 
however, only p^hy this role i^he has sorpe insights as regards the process of policy-making as such 
and knowledge about the structure of the educational sector and content of the, educational plans. 
(See separate paper. ) . ' . 

2.2 Identification of policy outputs ' » * 

Familiarity with the policy-making system described above is necessary when the various policy 
outputs are to be identified. These outputs may be expressed in writ^n document.s or only verbally 
stated. Some may be explicit ^nd clear policies, others rather vague. , Li many cases it will be 
next to impossible to find any policy^ guidelines at all except the fact that a decision is to be or 
already has been taken.*^ But one thing is for sure: goals always exist as long as the purpose of a 
programme or a project is to achieve something. It is this 'something^ that the evaluator n^ust 
identify. As the structure of policy-making varies between countries it is impossible to give any 
general indications as Jto where to look for policy outputs in gdvemmental bills, developing plans, 
committee background papers, plans of operations, legal documents, various protocols, budget 
presentations, etc. , etc. The only advice to be given is that the evaluator take the time needed to 
fiftd out what exists and what does nof. One may argue that it should be the policy -makers' job to 
communicate their policies in such a form that the evaluator never had to search for the informa- 
tion needed. But as ^his study tries to face reality as it usually e:»:ists such a recommendation- is 
seldom a solution to the problem to be solved. , * 

Instead we will consider the quality and relevance of the policy outputs in order to facilitate 
the rfext step of the goal analysis - the definfng of the goal elem'ents. ^ 

The qu&lity of the policy outputs only refers to how* weirthey are defined which, ol course, 
has nothing to do with the quality of Jthe content. . • , • 

The relevance of the policies must be determined with reference to the structure of the 
policy-making institutions. The reason of course being that in any field of public activities there 
exist various opinions of what ought to be the policies to be followed. If funds are to be allocated 
within, e.g. the educational sector, various interested groups such as the teachers, pupils, parents, 
officials at the ministry of education, political parties, leaders at regional associations, etc. , are 
interested to influeijce the decisions to be* taken. ^ The evaluator is of cour^e.ngt the ^ne to judge what 

* r ' • 4 
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are to be considered good and bad policies - his task is to consider those policies that are formu- 
lated in accordance with the accepted policy-making process of that particular country for this 
particular type 'of programme. JNTeither is he to introduce/hio-own political views into the anajysis. 
This may seem self-evident but how many times have not analysts introduced theor-etical models, 
without recognizing the implicit policies then applied. It may sound more scientific; to refer to 
l^reto optimality than to a vague governmental bill when analysing various expenditure patterns 
for^ducation. In terms of relevance for the evaluation the simple truth is, however, that a policy 
form\^ted by the government in the country concerned (e. g. that the regional distribution of new 



w schoolS sShould be considered) is far more relevant t han the policy ideas of a prominent economist 

\ • . 

from Italy in the beginnir;g of this century (which includes the idea that i;e distributional policies are 

not being conKid^3*ed). * 

\ * • * 

2. 3 Specifica^ion^of ^oal elements * , . o 

\ ,> . ' ' " 

The relevant policy i^ormation identified is now to be transformed into what can be called goal 

elements, tfiat is to sav formulated in operational terms from the programme evaluator's point of 

view. In the ideal case tne policy -makers, may present their guideliijes not in lengthy and vague 

statements but as precise goals. Usually, however, it is not only the task of the evaluator to 

identify policy outputs but he must also specify the operational elements in these policy declarations. 

A set of goals are usua^y^considered operational if all the individual goals are measurable 

in relevant units and if the individua^goals 'form a logical- aiiu-uiiambiguous whole in such a way 

that they can be achieved si mult aneousljr^ _l/ . The strong emphasis on quantifiable units has often 

stimulated analysts to consider only those.goals that can be expressed in operational terms and 

neglect goals that cannot be expressed numei;ically. To avoid this, one .may either stress that both 

operational and non-opet*ational goals should be^ncluded or widen the concept of ope rationality to 

include all goals that can be formulated in such a \^y that the goal fulfilment can be described 

either in quantitative or qualitative terms. \ 

We are going to apply the second definition of ope rationality which then will include a.quan- 

% ■ \ * ' 

titative, qualitative, distributional as well as executional dimension. 

- * * * ' "* 

The quantitative dimension 

* I 
The basic meaning of operational being measurable it is obvious that the quantitative expres- 
sions of the goals are of <5entral^mportance. The extent to which it is possible to quantify different 
types of goals and thus benefits are to be discussed in some detail below. 



J[/ TSero Pitkanen, Sweden Journal of Eronnmirg 1fl7Q. p/207. 
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The, qualitative dimension ^ ' , . 

* This should" be considered both as regards the qualitative aspects of the quantifiable units 
(e. g. the quantitative units 'student examined* can be qualitatively described in terras of minimum 
knowledge required tQ pass the exam) and for such aspects of the goal. that cannot be quantified 
.(e.g. description of what is meant by the goal 'a more democratic society'). 

What is important is that the language used when describing the quaUtative dimensions is 
as clear as possible. Oi>ly one interpretation should be possible. The socio-economic evaluator 
probably has a lot to learn from pedagogical research in this respect with its taxonomies describing 
in a systematic manner the 'qualitative aspects of various educational goals. A major research task 
would be to replace the so-called 'neutral* check lists with similar taxonomies focusing various 
macro goals related to educational systems. 

The following example shows a gradual specification of both quantitative and qualitative 
goal elements for a teacher training institut^e in Kenya lj\ - 

The first tentative definition of the purpose runs as follows: 

•*The goal of the' teacher training project is to remove the shortage of well-qualified 
teachers in a short time and at a low cost.by producing a number of well-quaHfied tea^Chers. " . 

Thii is a rather general statement without any reference to the Kenyan situation 'wh|ch 

^Iso lacks a time dimension. ^ . ^ 

^ % * ^ • . / 

Policy outputs presented in the project documents enable the analyst tv formulate the 
following: ^ 

"The goal of the Kenya Science Teachers' College (KSTC) is to contrib^ite to the removal 
of the shortage of Kenyan SI science teachers in a short time and at a low cost by 196B producing 44, 
47 by 1969, 98 by 1970, 128 by 1971 and from 1972 onwards 144 \yell-quaHfied. Kenyan SI science . 
teachers given a three-year course. *' " 

The qualitative goal element ''w^ll-quaHfied SI science teacher* is then specified and the 
definition of quantitative and qualitative goals ior th^-KfJTC project would then be as follows: 

"The goal of the KSTC project is to contribute to the removal of the shbrtage of Keny.an SI 
science teachers in a short time and at a low cost, 1968 producing 44, 47 by 1969, 98 by 1970,. " 
128 by 1971 and from 1972 onwards 144 well-qualified SI science teachers given a thr^e-year traio- 
ing. out of whom the academically superior will qualify for university entrance and the r^st, "bre- 
fe:rably all, -will serve as SI science teaclrors for such a time as ther^e is a shortage of SI science 
tf?achers i^ the country, trained so that they have academically reached a standard equivalent to 
Higher School Certificate in^their two major subjects, that they hjve a broad basis of scientific 



\j presented in a s'tudy by Olga Linn^: 'A sociological-pedagogical 
evaluation of Kenya Science Tea6hers College! Uppsala 1970, p. 23-27. 
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knowledge, that they have achieved skills in, positive attitudes to/ and knowledge of progressive 
teaching methods, that they will be able to function as innovators of these progrfsi?ive teaching 
methods with respect to their pupils and teacher colleagues and that, they have a good knowledge of 
social and economic conditions in their country so that the%' .are able to serve as well-informed 
opinion leaders4n their own local, communities. ' ^ 

ut should be nbted that this definition is not an official explicitly stated sjstem of quanti- 
tative and qualitative project goals -^but rather has been developed from interviews with the College 
staff and from documents available at KSTC in Nairobi). , » . if , * 

The difference in evaluative power of the last as compared with the first, specification of 
goal elements is obvious both as regards the quantitative and the qualitative dimensions. The rele- 
vant questions to be considered in an ex-post analysis are thus a^utomatically given in the latter goal 
fdrmulation and include for example: . ^ % 

'To what extent does the. KSTC project fulfil the §pal of an annual output of 44 by 1968, . , 
47 by 1969, etc. ? ' . . ^ * . . • 

, *To what extent and for how long will the SI science teachers graduated from KSTC stay in 

the teaching profession?' , 

'Does the'^KSTC project change the students attitudes toward more progressive teaching ^ 

methods?* . ^ ' « 

It may be interesting to recall at this point the difference between an evaluation - basedf on 
* •> ' / > 

specifications of relevant, goal elements —and traditional^a:5*ket-based internal rate of return eva- 

luations based on. income generated. In an ex-post evaluation it would be considered a failure if the 

trained teacher never became a'teacher but accepted jobs in"foreign private firms operating in the 

Capital/ With the latter method this may be considered a success* if the private enterprises paid a 

higher salary than the government could offer science teachers. . . 

The distributive dimension covers both distribution ovei* time and distribution betweeii 
various target groups. We will limit the presentation of* this dimenaion to the^ distributiorj over time 
and between groups. \ ^ ' * * • " ^ 

The distribution over time can be desc^ribed by a starting point (base-line), various check-, 
points during implementation and a specified time when the^final goals are to be achieved. , 

The base-linj*s sejrve the dual purpose of informing the decision-rmaker about, the present 

situation^ and thus improve the base for decisions as well as they enable the manager to ioiiow the 

goal variables over time as from the time before the programme was implemented. 

♦ * 
The checkpoints are particularly important for the management power of the goal, A goal 

stating that 1, 000 teachers" should be trained within a ten-year period is.^t operational until the 

yearly output of teachers trained is specified. This can easily be understood when. realizing the 

differences in implementing the following two different, output series (Alt, 1; 10 x lOp; 

Alt* 2: 5.x 20 5 x 180)." * * 
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The evaluative power of the time .dirpension of a goal is self-evident. No-one would con- 
side^r it irrelevant if the 1, 000 teachers were trained during a 5, 10 or 25 year period. 

In .spite of ihe obvious and absolutely crucial time dimensions base-lines and checkpoints 
are often missing and some educational policies are even lacking any reference to time. . A goal - 
without the time dimension can never be considered operational. 

The distribution between various groups (e. g. classified in terms of income^ age, sex, 
or geographical location) is equally, important but in spite of this often\neglected in mpst public 
goal formulations. Just as goals alwayi exiki, - at least in an ex-post sense - so Co target groups. 
When known, such distributional goals are usually either very vaguely formulated (eqqality is 
iniportant) or belong to the implicit goals that fall into contradictory pieces if expressed explicitly . 
Another reason for the lack of distributional descriptions between groups is that the present situa- 
tion is not known and thus the light of explicity will only reveal a lack of knowledge.- Finally an 
often heard ar^^ment for not focuping op, the distribution between groups v/hen appraising, public 
programmes is that taxes and other 'distributional' policies take care of these aspects. 

Any practical knowledge about reality will however indicate that the distributional effects 
of certain public investments neiiher can (due Jd its permanent character) - nor will (due to lack of 
political power) be directly linked* with other redistributional efforts. " C 

How can, for example, a farmer in the couflti^side be compensated for a government 
polfcy not lo consider the^distributiohal effects of such public services as education or health? 
Through a general redistributional policy of progressive taxation? Surely not. Conseguently the 
specific target groups for a public ihv,estmeht must always be identified in a goal analysis. 

The executional dimension . ' * . ' 

The concept 'operational goal' implies, as described above,, that the goal can be used as 
an allocative, management and evaluative tool. The executional dimension of the goal describes 
the extent to which the goaUh reality will be related to the allocative decisions to be talcen, the 
management process and! the evs^luati on worl^ to be carried out. 

Many political goals are expressio^is of compromises and may be the most practical way 
to avoid political cr iflicts. The goals then formulated may have no 'steering power* as regards the 
deVjBlopment on Ipwer levels.. 

In these cases the technique qf 'reve'aled preferences'' can be useful in checking to what 
extent the diecisions actually taken are in line with the policy statements, mad^e, that. is tc^make a 
cru'le but clear distinction between what c^n be considered real anddmaginary goala respectively. 
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2. 3. 1 Behf^fit measures actually appUed when evaluating educational programmes 

The identification of the benefits for a certain project is the same as.the description of the cont^rji 

of its, operationally defined objectives and goalp. In spite of this it may be interesting to indicate 

isonie benefit m^asures^s they actually have been used in various analyses in order to exemplify 

the possibilities' and Ij^mitations to, express progress data in quantitative and qualitative terms. 

These measures a¥e, of course,, only to be r^egarded'as examples of potential benefits. 

\, ' p ^ '\ / < ' ^ ' , ' . • ' ■ ' / 

(iK The principal measure pf productivity is the increased earniogs due t0[ 
higher education as isolated - as far as possible - from income effects 
.fjetermihed b^ native ability,, family backgrpurid, social class origin, 
' Voyk experiences, etc. . 

Even i£ we ^Cpuld 'isolate these factors and ail the reie;v^ant statistics ^ ' 
^ere available, it dan.be questioned, whether - from society's pQint of 
"(riew. - income levels- and productivity levels weres strongly correlated. 
I^rom the iridividtial's point of view the .earning as such is of course the 
criterion'vOf better opportunities and possible rise in level pf living 
pegardless'bf the productivity effect of the job performed. " ' / 
.Whenvcalculating.the productivity/income effects' of education the actual 
rate .of emploj^ment must be estimated (based on manpower plans) and 
, not school enrolment figures; drop-out rates, un- and underemploymejjt 

^ must be included in the analysis.. . . ' ■ ' \ 

(ii) Consumption benefits from education (the pleasure during and after 
\ * educatiort) is hard to measure and therefore often excluded. . * 

(iii) Social educational benefits include: ^ 

^ greater flexibility? and thus adaptability in a changing society due to 
> education (greater secu|*ity); , - 

- higher status and, perhapsT, more 'wanted' type of jobs; 

' - lower birthrates are often ^Sspciated with education as w.ell as 

o . improved health (due tb better hygiene). ' ♦ 

(iy) Indirect benefits 

- other persons earn more due to the education of 'some, e. g. employer 
subordinates, families (mothers can go out to work when the children 
are in school); , i ^ ' 

- less social cost for the public in the fiejid of police protection, social 
^ welfare, etc. (mainly relevant in .urban areas in industrialized 

. societies). ' . ' « 

* (v) * Institution building effects '\ , . ' " 

- deve:ldpnlent of an ingtitutionaLstructure adopting and forwar^dilig 
attitudinal changes with posiUv^ effects on the development prbcessi ^ 

- the maturity arid^tjuality of the teaching institutions can be measured , 
(e. g. by assessing the quality of the Inputs). 

(vi) Political benefits ' ' i 

- the inr\portance of a literate electorate; ^ * 
-lodging a national' culture and in many cases a common language; 

- modernizexi outlook both as regards those educated and their rela- 
>* * tiv^S'(e. g. parents) and friends. ' * : ' 
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j[vii) ' bistributionar benefits 

Education can bring about equality in oppprtunities and the distribu- 
tional benefits of providing education to those under the poverty line 
^are an extremely important policy measure to decrease the gap . 
between poor and rich, It should, however, be recognized'that the 
pjace of reoidence after completed school rather than the location of 
the'^chpol is most iniportant^^ A theoretical elementary education in^ 
a rural setting may just increase t^e 'unwanted urbanization and thus " 
unemplbymenf in the cities and be of no or negative effect for the cpm^ 
m^anities^where.the school was located. 



2.4 ' Structuring the goal eletnents 

The^goai hierarchy: . - ' 

VVe have now reached the* stage in the goal analysis where thfe relevant policies have b,een 
transforrfied into gpal. elements which are as operational as possible. The next step is to relate 
tHese goal elements to various policy ievels,. that is to say identify how ends and means are, related 
to^each-other.^ "Jhls can be done by constructing a goal hierarchy. 

* A goal hierarchy - or a means and ends chain as it is also (Tailed - can have any number of 
levels and is characteriz.ed by the f^ct that ends on one level beconxe means on another^ (See the 
figure below. ) , ' * , * , 
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tor practical purposes we wiJ use four level mod^l. These four levels are usually suf-. 
ficient whien analyzing a public programme and fcan be referred to as main goals, sub-goals, produc- 
tion targets anrf activities* ^ ^ ' ' . 
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The goal hierarchy can then be presented as follows: 



Main goal 



.... 1 






Sub -goal 




' Sub-goal 













Prod, target 1 > 
(phys. results) 
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Prod, target 2 
(pers. results) 


^1 


•Prod: target 3 
(inst. results ) * 



, 'ACTIVITIES 
. ' , ■l^ I \ -l i' 

*•*:•.. ■ COSTS ' • " ■ 

In constructing such a goal hierarchy .berth the 'top-down' and 'bottom-up' approach must be 
used depending on the extent to which the alternatives 4r,e a priori given. Again it is worth under- 
lining, the fact that the evaluatbr has to try to structure various goal elements into a goal hierarchy 
should not reduce - but stimulate - the policy -makers to do the same before the policy statements 
are formulated. This would be a most effective way o'f directly improving the consistency and 
clarity of almost all policy statements and stop the present presentation of isolated policy elements 
on various levels. - / ' 

Assume for example that the policy-makers formulating the main goal for foreign aid in an 

expenditure figure (e. g. GNP) had tried lO formulate some production targets. W^en realizing that 

spending^- ^regardless of use - hardly constituted any base for allocation decisions this main goal 

had probably been expressed as a resource base related tp some main more meaningful goals if 

one did riot accept the fact that, for example, 10, 000 monetary units (MU) spent on a Rolls Ro:^ce 

automatically are ten times better than 1, 000 MU spent on educational material. ' > 

This example may seem to be out. of touch with reality 'as no^ppJLiticians act without can- 
't' ^ 1- ' ' ' ' w 
sidering the use'pf the money* . But when realizing that most public policies are expressed not in 

berrfit but expenditure terms (X MU to the educational sector instead of stating the main benefits 

to be achieved through an ^expenditure of X HU) the abovQ. example is neither irrele\.ant nor^an 

undue wer-^implification. • , . , • ^ 

The goal hierarchy also enables us to apply the central concepts of significance and pro- 

ducthity described above. Planned and real signifiqance is then related main and sub-goals. 

while the numerators for planned and real productivity are to be found on the production (output) 

level. ' * . . * », • ' 

■ 51 , , • : . . 
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2*5 Analysis of linkages between goals on different levels / 
The logic of the goal hierarchy, Which implies that the various g^oal elements are 'sorted lA^on the 
appropriate levels, must be integrated with an analysis.as regards the lihkages between the diffe- 
rent levels. This is usually called an ends and.means or a means alyi ends test depending on whether 
.thj, *t9p-<iown' or *bottQm-u^' approach is used. ^ , 

dne way to un^lerstand and describe these linkages is to look upon the various goal levels 
as Separate subsys4^ms and then try to see how these systems are linked to each other. Ass'ume 
that we ar^to carry out an ex-post means and ends test. The starting point would be the input 

^ system specifying what inputs (in terms of e.g. inoney, mampower, goods and services) actually 
had t^een prpvided. The linkage between the input system and the project system is revealed through 
questions such as: Wa^s the money used in accordance to the plans? Could the manpower resources 

'be utilized? To what extent were the goods and services provided successfully absorbed by the 
project system? Wiat inputs were missing (lack of linkages)? Thg project system then tran3forms 
these inputs via various activities to outputs. The linkages between activities anci certain. produc- 
tion results is of course of extreme importance and is usually a problem to be studied Ly a subject 

' mktter specialist. - . , ■ 

The relationships between'the project system and the subsector to which the project belongs 
is of crucial intportance. . ^> . . . ' . o 

To consider only production and productivity has in many cases simply implied that the 
wrong outputs have been produced more effectively (improved teacher training for the wrong types 
of teachers, increased production of products without a market, etc. ). In other instances isolated 
produc^tivity analyses have stimulated the decision -makers to try to maximate a certain type of pro- 
duction without relatiog it to other related project systems and higher ends, A positive effect of 
stressing the relatiouships between means and ends is also that the subject matter specialists have 
to relate their knowledge and recohwendations to -a wider and more dynamic context instead of 
limiting their views, e.g. the project system. If this type of communication is improved the risk 
of the 'whi^e elephant' projects is reduced .and every subject matter specialist has to come out of his 
professional jargon in order to relate his job to the others* in order to achieve a common end. 

2.6 Analysis* of conflicts between goals on the same level ^ * 

"The governnient of any society may be viewed as composed of many competing^individuals with con- 
flicting goals . The last part of th*;; goal analysis will foQUS on the understanding and formulation 
of these conflicting goals. The whole process of resource allocation ^can of course be considered in 



1^/ J. Alcssi^ 'Implications of Property Rights for Government Investment 
Choices^ The American Economic Keview, March, 1969. 
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terms of various goal conflicts as ^ore resource? to one programme result in less, for another. 
The more Hmited scope of this analysis I's to discuss some possibilities and constraints in order to. 
•determine relevant trade-o/fs between e. g. stated production targets or sub-goals in order to 
create a base for the further analysis of the resource allocation problem. This is to be considered 
an-alternative to the present trend to neglect existing goal.conflictV at the polipy level and only con- 
""sider one goal at a> time. ' ' " " . ' ■ 

In terms of political slogans the goal§ for education, health and improved comm^inications. 

may then be formulated sepdi'ately in very general terms. Expressions such as integrated, unified. 
■ or balanced approaches nay' uowever give the public an impression of a consistent whole. Fr.om the 
* programme analysts' point of view such goals, resemble-jnore of a- wishing list than a base for any 

allocation decision. ■ , 

What is for example the meaning of "balanced regional development in the field x)f. educa^ 
tional facilities' . Is it'to be interpreted as if the differences between rich and poor regions in a 
country should be reduced (balance = equality) or remain unchanged on-a higher level (balanced = 
present distributional pattern)? • . ' ■ 

A simple systems approach can also, when analyzirrg relationships between goals on, the 
same level, be {iref erred to^a static expressioiflof a single goal as it helps the policy-maker and 
evaluator .to understand the dynamics ot the various goals." 

A major fask for the ev^luator is to identify inconsistencies within the goal, structure and bjj 
making them explicit hopefully improve the technical quality of the goal either by stimulating ppl^cy- 
makers^to reveal their preferences or by structuring the available goal information in a systematic 
way. , "v ■ ^' 

Not until the'se steps in the goal analysis are carried out can the identifica'tion,^ enumeration 
and valuation of relevant benefit expressib ns of a particular educational programme be focused §nd 
ar. ex-post evaluation - from society's point 6f A^iew - be useR.l,f<jr policy-makers and managers. 
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THE ROLE OF SECTOR STRATEGY AND SECTOR ANALYSIS IN THE 
PLANNING OF SWEDISH INTERNATIONAL DEVELOPMENT ASSISTANCE 



by L.O. EdstrOm 



^ < 



^ ^ » BACKGROUNb ' ^ * . \ 

The guiding principle of Swedish international development assistance is the recognition that {)lanning, 
implementation^ and evaluation of development programmes and pr.ojects is primarily the concern ^ 
and the responsibility of the developing country itself. Swedi^ asg^istance is country oriented, 
inasmuch as it proceeds from the needs, objectives and priorities of the r^ipient country.' 

1. For Sweden^s principal developm^t partners, SipA's long-tprm planning of bilateral , 
Swedish development co-operation takes the for^, of special assistance programmes, so-called 
country programi;hes.; This is the central planning document. It e/nanatesrfrom the recipient 

. countryjs own plans, priorities and wishes, and indicatep as well the volume and content of the 
Swedish development assistance to the country. The country programme consists of one des.criptive 
part and one containing recommendations, the latter forming the actual d^elopment assistance 
programme. * ^ f * ' 

2. The sector strategy and sector analysis are preparatory instruments which provide. the 
basis for discussions on the content and form of the development assistance programme., 

3. The sector strategy, the sector analysis and tSip sectot" plan are the three main concepts in 
any one sector. ^ 

4. The sector plan is synonymous with the all-inclusiveplans for the tdtal sector as expressed 
in the country's long-term plans, for example its five-year plan. It should include erivisaged 
development in all the sub-sectors within a particular sector. Swedish development assistance is 

in this context merely marginal as Sweden caii give support only to certain projects or certain 
adtivilies .within the framework of the sector p^an. The needs are often immense s^d far* exceeding 
the resources available.. For this reason the sector strategy and the sector analysis are ess^ential. 

5. The sector -strategy is SIDA's effort to clarify itd own concept of the sector. It is vis-a-vis 
the developing^partnersf in no Avay normative. It must not be used or interpreted as a directive for 
any one particular country as to what it ought to undertake Within the sector, nor as the soleicriteria 
against whicli tlie scope and nature of £he Swedish development assistance should be decided. 
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Priorities and wishes Expressed by the recipient country are. focal points.in the sector as 
elsewhere* ' " * * , o ' . 

^ The secto;f strategy is exclusively va document of the^donor country. In tlie first instartce. 

It IntiiC'ales how the donor country wou^d like to regard Jthe sector, the categories to be analysed 
witRin the sector, the type of infqrmation lieeded agd reasons forihe choice. .Secon^ily, a sector 

. strategy is necessary in those caVes where the developing partner'submits various proposals which 
it accords the same degree of priority or urgency, but where SIDA can only contribute to part of 

^hat is' requested because of the financial limit constituted by the target figure set for that particular 
couhtry. When' a situation arises whet*e selection of project/projects must be-made the priorities 
le^iablished in the sector strategy should apply . Thi^'dly, the developing country may often .reqoest 
advice pr assistance relating to the planning fof* the sector. In such a situation it as essential to be 

^cl^ear as to one's own position. Any advice^or help should nevertheless be given in the context of the 

-gener^.premise's of the country's own sector plan, , c > 

^* . '^^^ Rector analysis is a document where the sector plan ^and strategy are" brought together. 
It Is the central planning instrument for the sector. A strategy is only the prerequisite for the* 
analysis. . ' ^ ... * * « 

Analyses of the educational sector are conducted in those countries where development 
assistance iii* the field of education is requested. The analysis should be made against the background 
of the Sector strategy and contain one descriptive part and one in whicl^ comprehensive and;.detailed 
justif],cations are given for S.vedish contributions to the^country's development efforts. The sector 
analysis is conducted by the SlDA division .concerned and serves as a working document which, 
inter alia, is submitted to the area division forming a bas|s for the latter's. work' of programming, ^ 
development, assistance tp the fcountr^ ^ - - ♦ * 

V. 9^ The working out of a sector analysis consists of fiv^ main phases: 

(a) A draft description of the sector^^ based on av^il;stble documents such as development 
plans etc- , '* * , . ' « • 

' ^. . ' , ^ ' ^ t ' 

{b)'An outline for an analysis of the development and problems of the sectpr in the context 
pf'the description under (a). » . ^ ' ' 

(c) A listing of questions that are unclear in connection with (a) and (b). 

»** ' .1 ' 

(d) An investigation or study qf Jfacilities in the country, and visits to the authorities 
. concerned in brdWto identify needs. ^ * . ' 

(e) Pinalizatipn of the analysis. ' <^ ^ ^ 

The questions raised under (c) shwuld be submitted to the authorities of the country ir question prior 
to the field visit. , ' *^ \^ a''' 

8/ ^ The analysis caxi be conducted Dy the Development Assistance Office, SIDA Headquarters 
(Education division) by consultants, or a combination of thq, alternatives mentioned,, depending on 
what is most suitable in each case. 
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9. The descriptive part shquld, among ^h^ things^ include: 

(a) 'The importance of the sector in relation to the country's entire planning. 

(b) .The expressed objectives for the sector. , ^ 

^ m ^ ^ ' ' ^ * 

(c) The implicit objectives for the sector* (as indicated in the plans). * 
' (d) An accourtt of possible conflicting objectives. 

(e) The structure of the sector i.e. the educational system, lengt^i of education) age, 
levels, etc.- . , , ^> . 

(f) The size of the sector e.g. numbers of students attending a particular type of education. 

(g) yhe administrative structure of the sector (descriptive). * ' , « 

(h) The extent to which the sector^iainflu envied by/extemal* factors in society e.g. 
^economy, politics, cultural conditions, social structure,, language problems^ etc. This 
is an important pact otthe work because solutions to, the educationsJ. problems , are to a 
large extent to be found beyond the confines of the educational system and structure. 

(i) Target.grbups andirtrgions. 

• * * \ ^ 

.(j) Follow-up investments or supporting measures e.g. in relation to costs which the sectot 

' plan might imply in 10-20 ye'ars. 
♦ ' . ■ ' - ' * * - 

i (k) Financing and costs of the sector, ' , . 

(1) The safeguarding of the employment aspects in the sector plan. 
^ (m) Development assistance from abroad and its effectiS for the sector. 

(n) The ejo^rtence of evaluation within the sector and, if so, the methods used. ^ 

10. The analysis should, among other things, deal^th: * . 

, (a) Sfiortcomings - explicit and implicit - within the sector, its planning and administration. 

(b) Degree of realism in the planning. ^ . • . - . 

(c) Degree of inter-relatedness/ integration withm Uie sector. Do the sub-sectbrs and the 
projects within the^ sub-sectors together fohir. rt weli-coor'dinated whole? 

(d) The §xtent to which the objectives are achijved withiji^the sector. 

Je) Whether plans for the sector are realist'^ or merely formal i. ^. du Ihe real pripx^ies 
concerning finance, personnel aAd measured corresppnd to the priorities expressed in 
the sector plan? ' ^ * * , 

11. Recommendations for Swedish development assistance are made on the, basis of the 

country's own wishes. Preferably, each sub-aector for which assistance is requested should be 

'I , ' « ' 

regarded as a single unit, and proposals submitted should indicate in terms of percentage how much 

of the Swedish deyeiopmeiit assistancetintended for the entire sector should be allocated to e^ch 

» sub-sector (primary school, adult educie^tion, etc.)* 

Within each sab-sector an account should be given of tjie more important projects for which 

Swedish assistance might ]?e considered. The discussions on this assistance, however, chould be 

closely related to the sub-sectori the mentioning of projects servec as an^example 6nly* ^ 
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^The country's ydshes -oncerning tlie total resources heeded for a period of '4-5i years foir 
the projects within the .s.ub-sector should be indicated. The resources can be in the form of credits 
and grants comprising furHq, goods, consultants and staff.. . * 

Presented in this, form ^the recommendations, together with otlifij;;^parts of the analysis can 
form the basis for an agreement for the whole sector, n.b. within the frame of Swedish, develop- 
ment assistance to the whole country. Thus, an agreement with the country could encompass the 
total assistance to the sector, possibly with specifications for the sub-sectors whenever appropriate* 
The question of allocation between the different pfojects and thg type of agreed upon resources to t^e^ 
utilized would then be settled directly between the competent authorities of the recipient country and 
Si DA. A sector analysis would thus facilitate the possibility of the donor co^try to move away from, 
predominantly institution- or project-oriented Assistance t6^?L more flexible form which cpuld better 
serve whole sub-sectors and perhaps even an ehtire sector. 

The sector analysis should be submitted in English/ French to the recipient country for 
approval. ^ ^ . > ^ ^ 
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AN APiPROACH TO EXTERNAL EVALUATION OF TRAINING 



I. Ahman 



1, Evaluktibn systems a general outline 

WickstrOm 1^/ 6966) gives the following general outline for the relationship between different types 

^af evaluations: . * 

"The purpose of a project evaluation can be said to investigate whether a system has n;ioved 
towards the desired goals, and if this is not so undertake corrective action. The time-scope 
. ' for evaluation can be either pre-project (appraisal) where it usually is a question, of choosing 
batween alternate projects or sizes and compositions of one project; during project, where 
ic is desired to change activities (or goals) continually for better goal attainment; or past- 
project (ex-post), which is a final calculation of the degree of success of a project, A 
past-project evaluation can,, of course, be considered pre-project for new undertakings 
depending on the lessons learnt. • . , 

There are four requisites for a project evaluation: the system ha» to be knuwu, the goals 
have tb be stated explicitly, the goal attainment has to be measured and finally the goal 
atl^^inment has to be attributed to the project and its different parts. T>ie more ambitious 
approach, and one which is necessary for gradual adjustments of the project, is to assess 
the relationship between different parts of the projects, and goal attainment, i. e. to assess 
the effects of the project activities. In integrated projects ibis can be difficult and one has 
to be content with a 'black-box* approach, where the inputs are measured and the outputs., 
but nothing i^ known about the way the inputs produced the outputs. ' * . 

\% 

To make- the project workable, the main goals have to be^decompoged with an inci easing 
amount of specificify and 'detail, Th^e activities will then be designed and implemented. In 
'an after-measurement the gpals* attainment will be measured which sta^rts with attainment 
of detailed, specific goals for each activity and finally builds up to the goals for the whole 
project, as is shown in the figure 1 below. " « 



ERIC 



l^/^Bo Wickstrfim, 'Viewpoints on the evaluation of the CADU 
project (Ethiopia)?, 
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Figure'!. The goal - activities r goal-attainment sequence 

Main .^Sub- _^Production_J^ctivities J^arget ^ubgoal JVIain ^oal 

goals goals targets programs ^attainment attainment attainment 

Hence after what could be considered as the planning and implementation phase in the 
sequence the remaining 9tcps are introduce^ in the reverse x^rder i. e. starting at the lower goal 
level and ending up in the higher. ^ - . > 

Three fundamental concepts in connection with life-cycle of a project now remain clear; 
planning - iniplementation - evaluation. a . 

Figure 2. Planning implementation - evaluation 

Pla?^ . • Iinplementation , . Evaluation 

1 



Main- jjub- ^ 

goals ^goals targets 



^ ' — n ' — : — 

Production ^Activities JTarget Subgoal Main goal 



^ programs 'Attainment ^attafhment ^ttainnrient 

Novv, if th^ sequence is transformed into a V-shaped figure like the one in Figure 3. Th 
interconnection between the different. phases will be made more clear. 
^ ' l^igiire 3. Interconaection between the different" phases 

Plan , Outcome 



Main goals (MG) 



Sub goals (SG) 



Producticm target? (PT) 




Main goal 
outcome 



Subgoal outcome 



, * Activities (A) 



- — Target outcome 



Carried-out activities 



Actual costs 



Implementation 
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^ " The evaluation consists in comparing the plan with the outcome. Or to put it in another way 

you will always have, an input in the system through your: ^activities ^and you will always get an output 

or a Msult, But the output is not automatically to be referred to as the goals of your plan. First 

an evaluation (i, e. comparing outcome with plan) has to be carried out. Only if the comparison 

equalizes the ^tput with your goals you will have a complete goal attainment. ' 

More efficient evaluation systems consist of two complementary parts - a built-in evaluation 

.system - and a special evaluation system: . • - 

\ <^ • • . 

The built-in evaluation; system is a kind of management tool whioh will enable the decision 

maker to undertake immedla^ corrections in ttie course of the project. This system mainly leans . 

on reporting, as the device for dsjta collecUv'i. . However, the difficulty rests in the question, what 

is of value to report and wha,t is not^ ^ .r - 

The special evaluation system ^onsists of more thorough ch^ck-ups often in thelform of . 
field investigations like sample surveys^ observational studies or analysis of secondary data. The 
special. evaluation is not ecjual to the traditionai>>x-po^' evaluation i^ethod: . The special evaluation 
always is a tool to direct (control) the course of the^prpject towards established goals,_5vherea3 
most traditional ex-post evaluatio'js summon up the 'will* of the project so to Say with eventual 
benefit to new projects of equal shape^ but are too late to change the evaluated project. 

Figure 4* Evaluation systems' ' • N 
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MG 




Implementation 
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The interesting thing about the goal sequence is the inter-relatiohship between the different - 
goals and targets. The plan (as referred to above in the V-shaped diagramf can be systematized in 
the shape of a goal hierarchy where the different goals and targets form the 'building stones'. Idea.Uy 
the hierarchy could have a "structure similar'to.the one presented in figure 5. 

Figure s. Goal hierarchy , - .' • 

MG SG . PT A " 




A higher order goal will be derived through the attainment of lower order goals. There will 
be a logical sequence starting at activities lev^l (A) and ending up at the main-goajl level (MG). 
Goals will be interrelated but there wiU be no goal conflict . There will also exist a practical way of 
testing the different 'goals-means' chains in the hierarchy. (One such chain is specially indicated 
in figure 5). 

Simplified exampl'e of a 'goals-means' chain applied on a project prodding vocational 
training: - . , ^ . . 

^ A: to train lathing with a group of 50 trainees selected.from 

■ Why? . Why do we train etc. . . , ' 

In order to: 

PT: In May 1973 'at least 40 skiUed workers will leave the vocational training institute with 
such and such minimum level of knowledge of lathing. / 
TWhy? Why 40? Why that level? ' ' • . ' 

In. order to: • „ • . 

SG: Contribute to supply the need of skilled workers. It is estimate^d that an over-all need 
WiU be Of the magnitude of ... . out of which 40 will be educated at the institute. It is 
estimated'thaUhe minimal requir ement in knowledge will be. .. etc. 
^ , Why?. Why is this need quantitatL.'ely and qualitatively' at that specific level? 
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In order to: " - ^ / • ' 

MG: Contribi^e to the maintenance of existing industries. It is estimated that in cornparisoh 
to other alternative training opportunities, the present level technical know-how in 
the industries, difficulties of importing machinery etc. , , . the estimated yearly number 
of graduated skijled workers will accomplish the optimal efficient altemajtive, towards 
( ' -maintenance of existing industries. 

The. facts presented in the example could be debased but that is not the ppint;; The idea ip 
simply to show th^ logical interrelationship between the different goal levels andl^a way of establish- 
ing the interrelationship,' > 

Referring to eyaiuatioh it was mentioned that the goal-sequence was inverted at th^ end. 
The reason for inverting the hierarchy in the :evaiuation part of the s.equence is connected 
Wijh the fact that attainment of lower order goals (targets) does not automatically imply the attain- 
ment of the next higher order goal (or any higher goal) in the hierarchy. Reversely if the higher 
order goal i8_ attained the next lower order goal (or any ibwjer order goal) is likely to be attained. 
The following simplified example^ extracted from a popula^tioa project illustrates the point: 
main goal ' ^ » - less poverty in the average family, 

sub goal - less childreil per family; >' ' 

production 

target - access *o preventives for group A, 

(•A* equal to low income people in certain area; estimated population 'X'^individuals), 
activities - production and distribution of preventives, 

i 

Evaluating the project the planned activities will be compared to the activities carried out, . 
If accomplished activities equal.planned activities this does not per se guarantee that the production 
target has beeii attained. The distribution (even if it was properly carried out) could perfectly well 
result in group B having access to the preventives and not group A, B could for example be a 
middle-income group which better can afford using the preventives *and also be better motivated to 
do 60 because of their higher educational. standard, ' ^ ^ 

Next step in the sequence: if the production target hap been attained this fact does not 
^automatically lead to sub goal attainment. Access to preventives does not automatically mean that 
they are used in all cultures. 

Next step in the sequence: if. the sub goal has been attained this fact does not automatically 
lead to main goal attainment; less poverty. 

With some exaggeration the conclusion v;ill be, only when the highest goal in the hierarchy 
has been attained to we will really know if tlje whole exercise has been worthwhile. 

Or put it. another way. If the conclusion is that the population programme leads to increased 
poverty in the average family it should not be carried out, even if, for example, the result has, been 
fewer chilui*en, ' / , * ^ 
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In reality the causality is difficult to isolate and the situation is a. more complex one. But 
nevertheless the hiain poiqt remains to some extent and must be considered.. 
^* Evaluation systems - specific outlines for evaluation of education. 

Primarily this paper v^ll deal with evaluation of training. Hqwever most of the ideas presented .will 
also be applicable tp ediication. The distinction made between education arid training should be 
emphasized in this cohhection. Crawford 1/ (1961) defines trai^ning as determined by .a particular 
system in which the pupils carry out certain defined tasks. ' \ / \. 

The end product of training will be a particular on-the-job-bel)aviour. .In'the case of 
education th^ student is prepared to function in different typ.es of systems. Personal development 
and increased knowledge are often goals in connection with education while development of motor 
skills and attitude formation are-common objecj^tives fox* trainijig. ^ 

In order to facilitate the demonstration of a system for external evaluation the case of 
application to training will bemused. From the point of view of the evaluation sysiem it fpllows that 
the concept of training could be extended to include all forms of university education and alL.forms 
of primary and secdndar^- Education which are connected with specific aim for the society. The 
main exception being different types of education v(hich form the basis for other types of education. 

2. 1 The questions to be answered by th e- evaluation 
The usefulness of an evaluation system could be demonstrated by a sample of questions which could 
be answered by an appropriate uUlization of built-in evaluations and special evaluations. 

Example from an institute of secondary technical education: ^* 

(a) Goals * . j , . . ' " . 

MG: Contribute to inct*'ease the maintenance rate of operating machinery in 

the following industries ... 

» 

SG ; To supply the estimated need of technicians and skilled workers in' the 
* ' ' field of . ' 

X oPTr Quantitative educational targets 
' 1972 X persons trained 
. , ' 1973 y persons- trained , . - • 

^ / 1974 z persons trained 

. • ' ' Qualitative educational tar gets 

• Student performance. ' ? V 

Standards for the student's performance during each period of examination arc set. The 

natural examination period follows the end of each academic year and the end of the complete 

training. Standards set covering different dimensions of education which should be connected to 

the established goals* Some possible dimensions may be categorized, as: 



\ 
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Psychological Principles in System Development N. V, 1962, 
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^ - ^eoretical knowledge in connection vAth profession; ' ' • ' 
l''Practical knowledge in conne!ction with profession; 

• ^- ihnovative^thinking (attitudes, motivation, valueis); * ' : 
, ^ -^production-oriented thinking (attitudes,, motivation, values);. 

/ ' ' thinking oriented towards 'social relations in the group cfynamic situation 

at the industrial working place (attiitudis, i?lotivation, values): etc. 
\ ^ . . Teacher Performance ' \ 

'The pedagogical emphasis should be on: ' 

I / ~ modern teachi::g techniques using; group work^ and-.different au.dio-visual 

\ ' means to increase the motivation among the studients.' 

* - practical performance rather than theoretical knowledge. 

\,' - thinking in productive terms* v' ' 

^ - thinking in terms of coming social relations in the grpupiworking. .situation 

• " , • . pn the place of work, ' * - . 

A:. - to teach; ^ ' , 

* - to traiin teachers; / ^ 

* - to furnish, the schools wijh adequate equipment; 
. - tovbonstiruct the schools; - 

- to construct the school furniture and design the school interiors 
(ralDor^'atories etc. ), 

costs. ' ; ' ' - • • 

(b) Eyaluatiori ♦ 

The evaluation should make it ppssible iu get the ansvfers to the following questions: 
.On the Activities and Production Target Level: - 

' ^ , " '^r® there any delays in the Buil'ding program? What are the reasons? 

How can the difficulty be solved? ' Have the targfets for 'Furniture' and~ 
' ^Other Physical FUciHties! been attained? Equipment? 

- Wh^t is the actual student enrolment? How does it compare to the plan? 
What are the reasons' for deviations? How is the teacher trailing operating 
in con^iparison to the plan? , * 

- What is the student drop-,out rate? Why do students di'OR-out? What will 
hs^ppen with the ^rop-outs (negative effects of education)?' How can* , 
dfoproiit figures be decreased? . . 

-^How is the training program going? V/hat has actual been taught? What 
aspecjax>f education has been stressed? What effects have this had on the 
» sta^nts?. . . , ' ^ 

' ' * ' ' V ' ' > ' 

- What does the students attitudes towards the education look like?. What 
are the students motivation for studying and to the coming profession? 
Have any signs of development of .elitist attitudes among the students been 
seen? Undoubtly the students will be a privilegedwgroup. Will they cari^ 
with themL this outlook to the working- places? 
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- When oh the job, how relevant was the education^for the tasks carried out? 
Has the, practical training aspect had any effect? Even if trained in , 
productive thinking do the students still t^ink productive on the job? If he 
thinks productive does he also act pJ»oductive? In what way? Are the 
students innovative on the job? How do the students affect workers and 
other technicians on the job? What are the social i:elations? How . are 
th'e students looked upon? How are tlie workers and other technicians 
looked upon by the students! ©o the . students have any saying? Are they 
accepted as leaders? , How ar^ the students beihg used by the employers? ^ 
Are they underutilized, overutilized, or falsely utilized? What in the ^ 
training, in the students opinion,, was thie most suitable.subject? 

- Are ipoderi\ educational techniques and methods bein^ introduced? Does 
the student accept ^the tnethods? What complications have followed the 
introduction of the modern methods? . ^ ' 

- How is the administration (including teaching. - administration) of the. 
institute and schools working? Are there any changes in comparison to 
the set administrative structures? Are there any particular adminis- 
trative problems which have to b.e splved immediately or in th^ future? 

» • ' * • f fr • 

- How is the teacher training developing^ What refresher courses have 
been arranged? * 

] ' The evaluation design , . * 

An. elaborated built-in evaluation system wpuld tsike care of the answers to most of the 

questions in this connection* The danger with a built-in evaluation system rests in the possibility 

of over reporting. » ' , 

In order to complement the built-in evaluation some special evaluations, should be carried 

out. Particularly in connection with the utility of the education in the working situation some sample 

surveys or participant observation-type studies would be suitable. 

Special evaluations could also be used to follow the students during tlie training. In this 

connection an aptitude testing device will play an important role. 

On the Sub-Goals level . " 

Xhe evaluation, carried out should give answers to the following questions: 

r How many of the graduated students were employed immediately after the ^ 
graduation, how many ,(5 months after and how many 1 year after? I ^ 

- In what kind of work were the students employed after graduation? To 
^ what extent did the workjcorrespond to their education? Had the work^ 

carried out needed a different education? What was,the reason in this 
case that the student was 'wrongly* employed? Is there a decreasing ne6d 
for professionals in the field tlie students were trained in and an increasing 
need in a slightly different, or different occupation? Are there not suf- 
ficient educational possibilities to provide trained people for this different 
field or are there any other factors influencing the choice of job in a 
wrong direction? 

- What was initially the expressed motivation for studying at the institute? 
What was finally the expressed motivation for choosing the actual job? 
What future employment perspective is predominant within the group of 
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graduates and neyrcomers on jobs? For example is the technical education 
given at the institute seen as a short-cut to an academic career or for top 
positions in adininistration and industry^or is it seen as an opportunity to. 
be trained for productive technical work in the industries? 

- Are demand figures for technicians and skilled workers educated in the 
' field of • , • , (the fields of training) • • • • increasing or decreasing In 

whidh field is the demand increasing (decreasing) most? What major 
. . factors count for increased (decreased) demand? 

— Evaluation desigfi ^ 

- The studentif. could b^ followed-up at intervals after graduation by sample 
surveys^ (spe^cial evaluations), . * 

- Official demand figures from different industries coul(^be analysed 
(built-in evaluation and/or special evaluations). 

On the Main-Goal level - 

The following questions should be answered by the evaluation: 

- Has tHere been any change in thfj maintenance of production machinery and 
the equipment used? .HTave changes related to the availability of spare ^ 
parts occurred? Are machines operating at a higlier average capacity? 

Is this a result of diminishing tim^ lobt for repairs? ^Is repair working . 
more efficient or do the operators,^ow* how to handle the machines and 
JJie equipment, more rational? Are bresJcdowns in the production because 
of incorrect Operations or failing control syste*ns (gadgets) decreasing? 
Is this a result of better instructions given by skilled workers and 
^ technicians? In what way have they influenced workers and other 

technicians? 

^ Evaluation design ^ ^. 

^ - The individual in the working situation should beiollowed up. Persons 

responsible for technical production and production economics should be 
interviewed. Production results per production unit should be analysed 
(special evaluatipn). 

3, Outlines for an evaluation system for education (training) 

Often activities or even projects will be carried out with little or connection to the objectives 
that integrate the activities (project) to the needs of the society , This fias particularly been the ^ 
case of training which often has been looked upon as a separate specialization. It is therefore an 
urgent matter to perceive the function of training correctly and define the objectives'Of education as 
compared to the objectives of other activities which together form the complete system. 

We will now turn to the original hi^erarchy of goals (as presented above) and apply the model^ 
to the specific field of education (training). Relatively few components will be included in the model. 
Hence, it does no more than provide a schematic reflection of the realities of training, 

. J 3,.l Main-Goals - analysis of overall objectives 
Starting from the main goals in the society the role education can play as an alternative to attain 
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these goals has to be investigated^ The formulation of the main goals for education will ftirther 
r©quir\ an analysis of existing manpower^resources (knowledge, teachers etc. ) the cost of attaining 
different objectives and the material (school buildings, inventories etc,) available. 
' 3. 2 Sub--Goal8 . ^ < . 

(a) Analysis ofv,quantitative needs 

On the sub-goal level the number of people to be educated or trained for different purposes has to.be 
estimated. The starting point may be the capacity to absorb different professionally trained 
categories. What people have to be trained and the deadlines for the educational effort should *dso 
be formulated.in the goals. /What school premises will be. available? What possibilities for 
' on-the-job training will exist? ^ ^ . , 

(b) Analysis of qualitative ^eeds • 

The knowledge and skills required to carry oat the job. should be formulated in the form of objectives 
for training. A careful examination on the behaviour on the- job will result in a Job-descripti9n which 
should summarize what is needed to carry out the job efficiently and successfully.. So far job- 
descriptions have not shown t^e required precision required for a guiding instrument for the 
objectives of training. ^ 

^ In analysing the dutiei^ to be carried out on the job a rumber of questi<5hs should be put 

regarding:^ » ^ * . 

' ' , ^- How is the task carried out? 

- What instruments are used? 
cT^Why is it carried out? 

, « , ' - How well does he pat^ry it out? 

- What is the minimum acceptable standard for carrying out the task? 

3, 3 Production Targets 

Quantitative training objectives are formulated in accordance with the 
need for educated (trained) people presented on the cub-ggul level, the planned number of graduates 
per educational institution and per year should be stated in tl'ie production target. In order to achieve 
this output the number;of intakes at different levels per institution and year should be described. 

^ . ^ (b) Qualitative tr£:lning qjb^jectives entails a description of the desired . 

performances at the end of the training. In order to make the objectives/conamunicable the terms 
of competence the trainee must fulfill at th^ end of the courses must be described in an operational 
way. ' • ^ 

Mager 1/ (1961) sets out a number of fundamental requirements which shouiu:he fuKiUed to 
make the objectives clear. 



1/ Mager, R.P. "Preparing Objectives for Programmed Instruction". 
Fearon Publishers, San.Francisco 1961. 
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(i) Identification of the final behaviour which the* trainee is expected to 
produce. ^« - 

{ii)^Description of the acceptable standards of perfot*mance . 

(iii) Des<:riptioh of Me specifip conditions whlfch must be.prevailing when 
. measuring the performance. ' * * * . 

3.4 Activities , 

The foUowing'liiaJiirt^^ 

9 ..." 

. » ' (a) The course content. 

(b) Different media to be used. . * ^ ^ 

(c) Specification of ^ntering= performance of trainees. 
The course content " » 

The course content has to be specified on the basis of the entering trainees knowledge. 

. Different media .to be used ' ; ^ «» • 

Some questions must l^e answered in relation to this point like the adminisiration qf media and the 

cost of different type? of media. j 

5 . ^ Entering performance " - ' 

• • -* ' , 

In order to show the effect on training by using a performance differential the initial and final level 

of knowledge and skill must be demonstrated. The better the entering knowledge is known the more 

efficient the training can be carried out.. 
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THE EVALUATION OF THE KENYA SCIENCE TEACHERS COLLEGE 



1. Introduction ^ 9 ' 

The Kehya Sc4ence Teachers College (hereinafter called the KSTC) is a project supported by the 
Swedish Government for the training of science teachers for secondary schools in Kenya. The 
co-operation between ^he^goverhment* of Kenya and Sweden is based on a bilateral agreement 
covering the period 1965-1976, The main responsibility for the administratior\ and completion of 
the-nrojcc'v has been delegated, to the University of Uppsala by the Swedish International Development 
Authority (SIDA). * * o 

T)ie KSTC offers^ a study programme including academic courses^in Mathematics, Physi^^j, 
Chemistry, Biology, Geography and Industxjial Arts, as well as methods of teaching and teaching 
practi&e for students who have passed the East African Certificate of Education pr equivalent at the 
0-level. After three years of stgdy'at th^e KSTC the students are qualified to teach in Forms ^ - IV 
in their two main subjects and in Forms I - II in a subsidj,a; y'subject. At its full capacity in 1972, 
the College will hav6 a total enrolment of about 450 students and .produce.about 150 secondary -school 
teachers a year. <s / ' ' ^ \ , * 

At the request of the Univers^y of Uppsala ari.d the SIDA, Dr. Olga Linn4 commenced an 
evalyation study of the KSJC in February 1969. ' . ' 

The aim of the evaluation study is to make an assessment of the quantitative and qualita.tive » 
results ot the activities goi ig on at the KSTC and on the basis of these results to develop the present 
training, system furtbc; . ^ " ♦ . 

^n the planning and completion of this study. Dr. Linn6 has worked in close co-ogeraaon with 
the staff of the KSTC, repr^esentativjes of the University of Nairobi, Especially itsinstitute for 
Development Studies, and r^presentatjlves of the University of Uppsala. 

,2., P^^oject goals 

A detailed definitiqp of quantitative and qualitative project goals must for obvious reasons form the 
basis for any evalu^tio^ study, Xhe determination of the goals for the KSTC project has in the study 
been based upon facts given in interviews with the staff of the College and upon available documents. 
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/ - . ■ 

The goal and production targets of the KSTC project is to contribute to fhq remqval of the 
shortage bf Kenyan Si science teachers in a short time and ^t a low cost, by 1968 producing 44; by 
1969, 4?; by 1970, 98; by 1971, 128; and from 1,972 onwards 144 well-qualified SI science teachers 
given a^^three-year>training, out of whom the academically superior will qualify for university 
entrance and the rest, preferably all, will serve as 51 science teachers for such a time as there is 
a shortage of SI science teachers in the country, trained so that they have academically reached a 
standard equivalent tolligher School Certificate in their two majpr subjects, that they have a broad 
basis of scientific knowledge, that they have achieved skills in, ^'positive attitudes to, and knowledge 
of progressive teaching miethods, that they will be^kble to function as, innovators;of these progressive"* 
teaching methods with respect to their pupi\s.and teacher colleagues and that they have a good 
knowledge of social and economic coaditiohs.in tiieir country so that they are able to serve as wellr 
informed opinion leaders in their own local communities. 

It should be noted that this definition is not^ an official explicitly stated systeni of quantitative 
and qualitative project goals but'rather has Ijeen developed from interviews with the College staff ar^d 
from documents available at KSTC in Nairobi. • ^ / ^ \ 

- « 3. Objectives for Evaluation of the KSTC'Project 

?To evaluation component was built into the KSTC project in its initi'al planning. ,Hence no explicit 
indicators were set forth by the planners which could be used as a babis for an evaluation of the 
KSTC project. The purpose of this chapter is to specify how the^project goal definition can be used 
a£t a basi^ for generating criteria for, this evaluation. ' . 

The project goal definition sets forth desired quantitative and-qualitative changes to result 
from the project. The definition also includes explicit criteria for evalua/ing these results. Among 
the results set forth in the definition are the objectives of achieving the described quantitative and 
qualitative teacher- training results in a shoit time and at a low cost. It should be noted that this 
evaluation is sociological-pedagogical in nature aqd hence no evaluation bf the economic or cpst-time 
dimension will be included. This evaluation is primarily concerned with the effectiveness of the 
educational process of the project relative to producing the desired quantity and quality of teachers 1/ 

The following quantitative and qualitative project goalSii^jcluded in the definition are th^, 
objectives for this study: ^ - - • / 

^ (a) Quantitrvtive project goals to be studied :' ^ 

- • «^ 

1/ In any event if would have been difficult to perform an econoniic eValuatipn 
as cost and time criteria for the project have not been available* Hpwever, 
it. would of course be of great value in the planning jof future teacher training 
projects to include an economic evaluation component along, with the sociolo- 
gical-pedagogical ^valuation. 
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* " * (i) To what extent does the KSTC project fulfil the goal (production target). 

, of an annual output of by 1968, 44; by 1970, 98; by 1971, 128; and from 

1972 onwards 144 SI science teachers? 

* Ui) To what extent and^for how long will the SI science teachers graduated 

from KSTC stay in the teaching.profession? . * ' 

(b) Qualitative project goals to 6e studied \ ^ 

(i) Does the KSTC project chahge the students^ attitude towards more 
/ ' progressive teaching methods? 

' . (ii) How dural?le and consistent is any internationalization of a more 

positive attitude towards progressive leaching methods in the fprmer 
KSTC studehtsMeaching? ' ' 

^ ^ (iii) Does the KSTC project give the^ students the capacity to become wpll 

\ informed opinion leaders in their local communities? 

(iy) How durably and consistent are the former KSTC students in any 
* ^ < fun^ction they serve as opinion leaders in their communities? * . 

(v) Does the K^TC project prepare the students for the role of innovators 
of progressive teaching piethods withjrespect to pupils and teacher 
colleagues? 

(vi) How durable and consistent are the former KSTC students in any 

function they serve as innovators, of progressive' teaching methods with 
respect to pupils and teacher colleagues? , ^ 

In addition to this^ a substantial part of the study consists of descriptive data which can ' 

serve as a background to the above-mentioned evaluatioij objectives. Among the complex of 

questions within this descriptive part of the study, the followirfg questions may be pointed out as ^ ^ 

important: * " 

(i) What are the characteristics of.theKSTC students^ social, economic, 
and educational background and what are their motives for choosing 
/ ' • ihe teaching profession? . 

(ii) Will there occur any change in the students' assessment of the 

teacher's social status and their own motives for becoming a teacher • 
during their three years at KSTC? ^ . ' ' , 

^ (iii) What possibilities fgr acting as innovators and opinion leaders will the 

former KSTC studeAts have when they are posted in a secondary 
• . school as an SI science teacher? 

Thus it can be seen from these objectives that this study will be based partly on changes 

in the KSTC studeyiVs during their college time and partly on what happens to the forn^ier KSTC 

students when they are posted as professional science teachers in their local communities. The * o 

data froip the analysis of the professional situation will be related to relevant criteria xn the project 

goal definition in order to assess the impact on the students of the KSTC educational process. Thus 

internal criteria stated in the project goal definition are to be used in the evaluation of project ^ 

rcsi^lts. Furthermore, the study intends an evaluation of the total impact of the educational process 

at KSTC, including the factors of teacheirs, buildings, and equipnlent, course structure and 

•educational policy. < . ^"X. * " ' * * 
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4. Design of the.Evaluation Study of the KSTC project 

^ Administration of the Evaluation Study 

This evaluation stifdy is designed to run from 1969 till the end of 1972. The study is administratively 

divided into three phases: v > . ' ' . * " 

P<iasel:- 1969 , ^ , 

. Phase II: 1970-1971 ^ . I ^ ' 

* - ' ' ' -Phase III: 1972 

The design and the instruments for phases I, II, III were developed in Nairobi where the 
evaluation project leader was stationed from February''l969 to May 1970. The cojllection and analysis 
"^of the data from phase I, the report of phas.el and the outline for the collection of data for phase IT. 
and III were also prepared in Nairbbi. * 

The. evaluation of the quantity and Quality of teachej^-output is carried out by relating 
measurement data to infernal quantitati^ and qualitative criteria for goal fulfilment found in the 
lytoject goal definition. No attempt has been made to use external criteria, such as comp^aring^ 
quantity and quality of the teacher-output from KSTC witti the output from other .teacher training 
institutions. . ' ' . 

It has been suggested that- a comparison could be made between exam results, of two groups 
of secondary schqpl students,- one . group taught by KSTC -trained teachers and the other by teachers » 
.trained at pther institutions in order to get an external measurement of the quality of the KSTC- 
graduated SI science teachers. , ^ 

It has not been possible to carry out such a comparative study during 1S69, as the first 
group of students ^graduated from KSTC and thus did not start to work until 1969. It might be 
possible to try a comparative' approach in 1971 when KSTC has produced two groups of teachers who 
have been cut teaching one year or more, i. e. students who graduated in 1968 and 1969, or in 1972 
when KSTG has.produced three groyp^ of teachers^who have been out teaching one year or more. 

Hd>yever, the (ase of exam results as. a criterion of quality of teaching seems, dubious, as 
tjie exam is still mainly related to a traditional educational system. On the other hand it may be 
argued that the mere fact of a conservative exam system makes such a comparative study ^jeally 
interestitig as the teachers certainly, arp supposed to teach within the existing educational framework. 
But, even if exam results were selected as a criterion, such a comparative approach will lead to 
certain methodological ^fficulties. For example the problems of: 

(a) Selecting a criterion for how long a time (1, 2, or^3 years) a KSTC -trained teacher must 
have been teaching a certain group of students to make an impact on their e^fam results. 

(p) Selecting a criterion for how many subjects {1, 2, or 3 subjects) a.KSTC -trained teacher 
must Mye been teaching a certain group of students during a certain time'ci^ 2, or 3 years) befqre 
their exposiureHo progressive methods would be 'Sufficient to have an overall impact on their exam 
results* . , ^ . , * " . 
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(c) Selecting-a realistic sample size of KST.C -graduated SI science.teachers, given that, ftfr 
example; a sample of 10 KSTp teacliers at 30 students per class will involve testing SOO.stMdents. 
50-KSTC teachers will, invol^je 1, 500 Ertudents, and 100 KSTC ^eacher^ 3, 000 students, With compa- 
rable-sized control groups the.total number of students required becomes 600, 3,.000,* or 6, 000 
respectively, obviously totals difficult to obtain and expensive to test, . . , 

<d) Matching teacher group^ - it is difficult to obtain SI science teachers not trained-at 
^ KSTC but trained at other similar institutions' who have been. teaching the same subjects'to a*similar 
group of students a similar amount of'time. . • . 

(e) Keeping constant other variables such as .aided/.unaided schools,' iirban/ rural schools, 
equipment-Qf the schools, pt.aff at the S(ihools, etc,, which may confound the results. . ' - 

Given adequa:te funds, somfe of the above-mentioned methodological ;difficultie.s might be • 
surnftunted, but (d) and (e) can probably not be overcome sufficiently to make it possible to 
confidently draw conclusions about the quality of atudents produced by KSTC graduates vis-S-vis 
those produced by teachers trained ait other institutions, " .. ^ ' 

. . , , , . Populations ' • ' i 

Por the three phases of the evaluation the following groups of "students will be studied: 

Group 1: newly admitted students at KSTC. / > 
' Group 2: third-year studebts (i. p. the students in the^ir third year at KSTC). \ 

9^W'3- graduates (i^e'. the students graduated fromK^TC). 

During the eval>iatioq, phase I. 1969, th^se three groups of students will, of course, consist 
of different p^^^ in lat^r phases the same students will be measured at three 

^ different intervals, namely whien they are newly admitted, when ,they iiave studied for thre^-years at 
KSTC and then some time after graduation from KSTC. ^ , ' ' 

In addition to these three groups of respondents, a fourth group has been studied during 
phase S^. 1969. namely: ^ 

Group 4; ftitors atlCSTC. ^' . ^ . 

Relations between testing methods and populations 
The evaluation of quantitative proj eel goals will be carried out by record-keeping. The evaluation of 
qualitative project goals will be carried out through group tests, mail inquiries, and individual 
interview.s. ' ^' 

The students who are currently studying at KSTC will be tested through a group admin- 
istered test, through this method data will thus be collected from the groups of the newly -admitted . 
students at.KSTC and the^third-yearkudents at KSTC. \, 

The same methods of data collection will be tJii^ed even for the group of tutors at KSTC* 
Mail inquiries and to a certain degree individual interviews will bp used on the graduates of 
KSTC. ^ ... , , ' 

.10 ■ ■ . ' 
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Instruments ' * . • 

There are six major measuring instruments for the evaluation of. th^ qualitative project goals. . 

(a) Background questionnaire 

r A general description of the students* social, economic and cultural 
background; 

* " ' ■ ■ » 

- the students' motives (or status -estimations of and ex^ '^iences of the 
teaching jprofession. 

(b) Attitude scale 

- The students* attitude towards progressive/ traditional teaching" 
methods. - • v . . 

/ ^ (c) Knowledge test 

. / , - The students' knowledge of cultural, political jand economic matters 

primarily ia East Africa but also in other parts of the world. 

(d) Adjustment questionnaire 

- The students' adjustment to and social status. in their new occupation!; 

- a general description of equipment and social, contacts in their new 
- / ' school. 

/ . (e) Modified adjustment questionnaire 

- Same as (d)-l, but including more data on the SI sci^ence teachers^ 
conduct in their teaching and their role as possible opinion leaders in 
their local communities. , ^ ' 

(f) Extra questionna;lre 

- This qucfstionhaire £*esembles the background questionnaire and thus 
takes up questions about the students^' attitudes to and motives for 
becoming a teacher as well as data about th.eir estimations of the 
status of the.teacher profes^on in Kenya. 

Relation between quantitaj:ive and qualitative evaluation objectives and 
. . 1 ' " population - ' - ' * 

(a). Quantitative objectives * 
Cybjective l ;.To what extent does the ?CSTC project fulfil the goal of an annual output of bv 1968, 44; 
by.l969jj, 47;% 1970, 98; by 1971, 128; and from 1972 onwards 144 SI science teachers? 
Instrument: Record-keeping. Reliabl'e records are kept atKSTC. * 
The populations: This objective is studied through comparisons between the same population, 
different years^* . . , . ^ ' \ 

^ • The newly admitted students 1966 - the graduates 1969, 

- The newly admitted students 1967 - the graduates 1970, 

- The newly admitted students 196j8 - the graduates 1971. 

- The newly admitled students 1969 - the graduates 1972. 

Objective 2 ; To what extent and for how long will the SI science teachers graduated from KSTC stay 
in the teaching profession? , ^ 

■ . . • \ 74 . ' ' • 
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Instrument: Record-keeping. Four approaches to reach the graduates will be used: ^ 

(i) Contact with Teachers* Service Commission in,Nairobi for checking on ^ 
the graduates* postings; . * 

% (ii) infoi;mation from the graduates on changes in their postings oi; careers; 

(iii) contact with the graduates* headmaster at the school where the graduates^ 
' were last known to have been teaching; 

* Jiv) tracing of graduates through sending out interviewers to the school where 
the graduates were last known to have been teaching. ^ 

The populations: , ^ . / 

- the graduates 1969* contact taken 1969, 1970, 1971 and 1972. 

- the graduates 1970,. contact taken 1970, 1971 and 1972. ' ^ - 

- the graduates .1971, contact taken 1971 and 1972. 
the graduates'1972, contact taken 1972. 

' (b) Qualitative objectives (i) to (vi) and (i) to (iii) ' 

In order to measure the qualitative; project goalsi three methods will be used: group tests, mail 

inquiries, and individual interviews. 

Objective 3; Does the KSTC project change the students' attitudes towards more progressive teaching 
methods? . * ^ 

Instrument: The'attitude scale measuring attitudes towards progressive and traditional teaching 
methods. • 
The populations: 'This objective is studied through comparisons in two diffef^ent ways: 

r different populations r the same year .a «. - 

In phase I, 1969, through a comparison between the newly admitted students 1969, the 
third-year students 1969, and th.e graduates 1969. 

<i 

' - the same population - different years 

Through comparing a group of students in two different roles, i. e. as newly admitted 
students 1969 and as third-year students 1971. * 

Objective 4 : How durable and consistent is any internalizatipn of a more positive attitude toWards 

progressive teaching methods in the former KSTC students* teaching? 

Instrument: The attitude scale measuring atfitudes ^towards progressive and traditional teaching 
methods. 

The populations: This objective is studied through comparisons in two different ways: 

- different populations - the same year ^ * 

In phase I, 1969, through a comparison between the third-year students 1969 and the 
gradultes 1969. ; . 

• o - the j3atne population - different years , , \ ' * 

Through comparing the same group of graduates.in different years., , 

Objective 5 : Does the KSTC project give the students capacity to become well-informed opinion 

leaders in^their local communities? 

Instrunient: The knowledge test on cultural, political, and economic matters primary in East Africa. 
The populations: This objective is studied through comparisons between different populations , the 
same 'year . ' HWf " '\ 
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Objective^ : flow durable and consistent are the former KSTC students in any function they serye as 
opinion leaders in their local communities? ' ^ 

Instrument; The modified adjustment questionnaire administered from 1 September 1970 to 1972. - 
the populations: this objective is studied on different populations pf graduates from KBJ'C in their 
secpnd, third, respectively fourth year as teacfeers. No comparisons are made between the groups. 
Objective 7 ; Does the KSTC project prepare the* studentsf for the role of innovators of progressive 
teaching methods with respect to pupils and teacher colleagues? « 
Instrument: The adjustment questionnaire administered in 1969 and the modified adjustment question- 
naire administered in 1970 through 1972. • . ' • ^ 
The populations: This objective is, studied on different samples of graduates from KSTC in their 
first year as teachers. No Comparisons are made between the samples^ 

Objective 8 : How durable and. consistent are the former KSTC students in any functi6n they serve as 
innovators of .progressive teaching methods with respect to pupils and teacher colleagues? 
instrument: The modified adjustment questionnaire administered in 1970 to^l972. 
The populations: This objective is studied on different populations of graduates from KSTC in their 
second, third, respectively fourth year as teachers. No coniparisons are made between the groups. 
Objective 9 : What are the characteristics of the KSTC students' social, economic, and educational 
background and what are their motives for choosing the teacher profession? 
Instrument: The background questionnaire administered in 1969. 

The popdlations; This objective is studied in phase I, 1969, on the newly admitted students to KSTC 
1969. ^ . , , . 

Objective 10 ; Will there occur any change in the students' assessment of the teacher^s social states 
and their own motives for becoming a teacher during their three years at KSTC? 
Instrument: The extra questionnaire, administered in 1971. 

The population: This objective is studied in phase 11, 1971, oa the third-year students at KSTC 1971 . 
Once again it can be noted that we will have a longitudinal pattern for these students as they will have 
been measured in two different roles, i. e. newly admitted students 1969 and third-year students 1971 
Objective 11: What possibilities for acting as, innovators and opinion leaders will the former KSTC 
student's hive, when they are posted in a secondary school as an SI teacher? 

Instrument: The adjustment questionnaire administered 1969 and the modified adjustment question- 
naire administered in 1970 till 1972. - 

The populations: Thief objective is. studied on the following groups: 

- graduates 1969 - measured in 1969 till 1972. , 

- graduates 1970 - measured.in 1970 till 1972. 
, - graduates 1971 - measured in 1971 till 1972. • ^ 

^ graduates 1972 - measured in 1972. 

No comparisons are made between the groups. 
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To help suminarize this methodology a diagrammatic outline is presented on the next page. 
^In order to make, the scheme easier to read, the evaluation objectives have been omitted as thiese 
cannot be related to a certain population alone or to a certain instrument alone but have to be related 
• ' to various. population^ and instruments during the years 1969 - 1972. 

\ *; • ~ . . ^ Background variables • . 

• The background variables used within different groups are the following: ^ , 

, , • *. - different grades of achievement at'KSTC; 

- different attitudes to more progressive/traditional teaching znethods; 
-,^^^erent postings after graduation fromKSTC (city schools/ country schools); 
"-'different ethnic origin (African/ Asian, citizenship/jipn-citizenship); 

- male/female; » . * T ' 

- different divisions dn Cambridge School C ertificate or East African C ertificate of ,^ 
Educatfon. ^ • . ^ ' . _ 

. 5. ^Report of Resulti§ ■ . ^ ^ 

, Evaluation objective 1 . ^ ' 

• In 1968 the output was 41 graduates, drop-out 3, and in 1969 46 graduates, drop-out 1. It can thus, 
be said that the KSTC prpject has to a great extent fulfilled the o^tpu^t goal for 1968 and 1969 set- 
forth in the project goal definition, ^' , ' \ ' - \ ^ ^ \. 

Eva^uation objective 2 . ' '\ ■ 

In the first phase report the results related to this evaluation objective ^re fr.om.only one group of 
graduates,' the 41 students who graduated in December 1968, and who started teaching in 1969. In 
September 1969, 33 graduates, 80 per cent, were working as SI teachers, 7 graduates, 17 per.cent, 
were siu^ing at. various universities; and oae graduate had j.oine^-private business. Eight kraduates 
have tliu.? left the SI teacher career eight months after graduation, but since 7 of them have taken up 
^ university studies after recommendation from the College, it can be said that the KST^ project so 
far to a great extent hais fulfilled the criteria set fort]i in the project goal definition ijx the sense that 
97 per cent of the 1969 graduates havfe behaved in accordance with the criteria. 4 . ^ 

^ ' ' ^ Evaluation objective 3 - ^ ^'^ . * > ^ 

Op the basis of these results of the attitude tests it. appears that the KSTC project has contributed to 
a change of the students' attitudes ia favour of more prbgressiye teaching methods during their 

* three-year study at the College.- o * , \ , • 

/ ^ Evaluation objective 4, • * . ^ ^ * 

On the basis of the results in this and previous sectionUt is possible to tentatively conclude that the 
College contributes to ,i change of the students' attitudes in favour of more prpgressive. teaching^;) 
niethod^^ anji that this chsmge, remains after half a year of teaching. 
' * . .V , , Evaluation^Objectivfe 5 ' ' ' . . 

It weVa that the studies at KSTC do not influence the students' knowledge of cultural, political, 
and tccinomic taatters in East Africia. This conclusion, however, is tentative as we have stHj^iied 
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Table 1. . Design iorthe evaluatidn study 6f KSTC, 1969-»1972 
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two different groups of students. The results might nevertheless be taken to suggest that the project 
does nof^fulfil the goal of increasing the students' knowledge of East African cultural, political ^nd 
economic matters - i. e. to make the students more 'well-informed* about the society in which they 
are gping to act as teachers and possible opinion. leaders. \^ 

Evaluation objective 6 ^ - ^ ' 

Study not yet completed: ^ . - 

Evaluation objective 7. ' ^ 

According to these results, based ori'the subjective estimations of the graduates, it appears that the 
KSTC project has succeeded in preparing the students for the role of innovators of progressive 
teaching methods, at least in the sense that the graduates are. actually using these methods and that 
their pupils and teacher colleagues on the whole seem to welcome this way of teaching. 

It should, however, again be noted that the results are^ased on the subjective estimations 
of one group of graduates. 

Evaluation objective. 8 
Study not yet completed. / * 

Evaluation objective 9 r ' 

A description of the KSTC students' social, economic and educational .background. 

Evaluation objective 10 
Study not yet completed. ^ o 

Evaluation objective 11 ^ . * 

It appears that the graduates, particularly those who teach in country schools. IWe under quite 
severe qonditions. which migM make it difficult for them to function eSffectively as teachers and 
hinder their ability to act as innovators of progressive teaching methods. 

It might, however, be noted that'B3^er cent of the graduates state that they like working as 
teachers extremely or rather well and that, this group consists of about the same per cent of 
graduates teaching in city and country schools, 65 per cent arxd 60 per cent. 

6. Concluding remarks 

At this stage of the long-term ev^uation study it iv only possible to draw tentative conclusions. The 
data and results discussed aboye indicated that the project is successful in its major goals of 
producing the desired number of teachers and that these teachers are so far remaining in the teaching 
profession. Further, the project appears successful in changing th^ students' attitudes in favour of 
more progressive teaching methods. In addition, the teachers appear to use the progressive teaching 
methods employed at the College despite the often poor equipment of their schools.. The College, 
however, appears not to be notably successful in irriproving the students' general knowledge of their 
cultural, political and economic milieu, which knowledge would probably enhance their ability to fill 
their likely rple as opinion leaders in their local communities. The data also indicate that many of 
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the graduates are somewhat dissatisfied with the lack of amenities and social contacts in their new ' 
postings. To meet these problems it might be appropriate for the College to introduce periodic 
workshops discussing current affairs. In addition, a seminar at the end of the three years could 
brief the students on some of the p^ractical problems they will face as new teachers. Recent 
graduates should participate in these seminars and issues such as those examined in this study could 
be .discussed. « 
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Annex II 



THE EXPERIENCE FROM FAMILY PLANNING TRAINING IN CEYLON 



During 1965 the Syr e den -Ceylon Family Planning Pilot project was transformed from a clinical pilot 
project in two villages into a nationwide governmental programme to include about 600, 000 families 
to accept and practise family planning in the course of a ten-year period. With this in rrdnd, a 
training programme commencing in 1966 and continuing until 1968'has been drawn up for all 
fceylonese health personnel (Medical Officers of Health, Public Health Midwives, Public Health 
Nurses, Public Health Inspectors). During the experimental period^^from 1958 to 1965, the activities 
ofthe Swedish pi'ojcct were of a medical-gynaecological nature and the Ceylonese pej^sonnel 
employed in these activities were given training in family planning. At the start of the governmental 
programme, early in 1966, *the training applied during the e^erimental period was retained - more 
or less - unaltered. Since this organisation had not been preceded-by any kind of systematic planning 
of training activities such an organisation being based rather on common sense' prgdictions of what 
the various groups of pupils needed to be taught, the obvious require^-nent was to evaluate the suitabi- 
lity and efficiency of the training given. 

This evaluation ^as carried out at the instance of the Swedish International Development 
Authority (SIDA) during the periods 23 April-27 May, YNovember-13 December 1966 and 14 June- 
30 July 1967. The main purpose of the evaluation was to find out to what extent the training 
objectives, methods and supervision as well as the content of the c6urses, were satisfactory and m 
what respects changes were called f^r in order to render the training successful. The preliminary 
phase of^the survey - when information was collected through participatioh in the ordinary work of 
the project, stucfy visits to the two training centres at Bandaragama and Point ^edro as well as 
through intervijBWs with course planners, teachers and trainees - showed that there was gooi call 
" for an i^mprovement of the efficiency of the training, and a seri^^ of changes was duly effected in 
collaboration with the Ceylonese teaching staff. The evaluation was concerned with the training of 
the following groups. Public Health Nurses, Public Health Midwives and Public Health Inspectors. 
Training courses for doctors were excluded on the advice of the Director of the project. 
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1* Evaluation ' ' ^ - 

This was primarily effected through two written tests, identical in^content, which were given at the 
beginning and end of the course. The purpose of this evaluation was to show that the trainees had 
increased their knowledge as a.result of the course and to provide them with a collective feedbaclCof 
the information gained during the cour/e.' The last item of the course dealt with these tests and 
included a discussion^of them. However, the teachers were not informed of the results. 

One of the tests (test 1^ consisted mainly.of items ot the essay question type (see Annex III), 
while the second test (test 2) consisted of four multiple-choice questions and five questions .of the 
-short-answer type. . ' « v , 

One point ^as scoued for every correctly answered question. The results were summarized 
for each trainee in the form of a total s^core for the pre-teij^t and.post-test and the difference between 
the two. In order to obtain a more compact expression; of the pre-test and post-test results, 
averages wei-e calculated for ^1 the courses held in 1966. The following tablevshows the, averages 
for Test 1 on all courses for mid\yives, nurses and inspectors. * . . 





Pre-test * 
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Gain 
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Nurses 


5.37 


8. 18- 


'2; 81 
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Inspectors 


5,86 


8.^1 


, 2.85 


71 





Average on Test 1 for the diffe;:»ent categories of Health Personnel at the Bandaragama 
training .centre during 1966. * * ' * 

'A,^' results of Test 2 v^ere similar. One is struck by the high pre-test level, which is - 
about half the n^aximum score or^ove^. This implies that the gtoup Iiad already attained 50 per ceAt 
of the objectives measureu-by the test fcefore att^ending the course. In order to show something of 
the pre- and p6st-test levels for all items,, the average for each item has ])een calculated and 
d^ipressed as a percentage ;(see figures 1-3). ' * 

Averages equal to half the maximum number of points obtainable aad over were noted for 
items 1,^ 3, G and 8. Nearly everybody answered item 3 correctly, and item 6 also has a rel'atively * 
high corf'ect-responsdT frequency. 

1 ^ 'y^® midwives showed a relatively slight improvement between the pre-test and the post-test. 
The possible gain amounts to 5. 17 but the actual gain only amounts to 2. 18, One should, however, 
riot he too precipitate in regarding this as sT measure of the effectiveness of the course. The majrking 
of various items left ascertain amount to be desired, added to which the tests only covered a limited^ 
number of the training objectives. But there is every reason to suppose that the evaluation-.left a ' 
certain amount to be desired. o " ♦ 
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2, Sumniary of the principal faults of the training 
The main faults of the training are suniniarized below in ten points, all of which were regarded , 
crucial to the part pWyed by the training in making the target attainable'. It should be emphasized 
that these points are to be regarded as a summary ^of the diagnosis made 'during the survey, and nbt 
as, a criticism of the training. - * c ^ ' 

(a)^Job descriptions regardinglhe tasks of the health personnel within 
family planning were completely lacking. The.tr^ining prograhime was 
indirectly based 9n the ideas of these tasks which had gradually been 
evolved duriiTtg.thS pilot project, * 

Certain topics were too theoretical and unnecessarily comprehensive, 
and complex in relation to the time available for the training and the 
, , \ nature of the tasks. • * / ' 

/(Q) 'There Vas a relatively large and unplanned.overlap betwef differ^t 
lessons. '.^ * ' * ' * 

(d) The time allotted to group work was^ i^adeguate. ' - 

(e) Teaching ai^da were to a very great extent lacking. The blackboard was 
the^principal resort on all occasions. 

^ ' (f) The field work (joui^neys included) took up no less than, about 46 per cent • 

of the total duration of the course, but. its training effect wasoin noting 
like the sahae proportion. * * * 

(g) The evaluation was designed to show that the trainees had increased their 
knowledge and to provide a certain feedback of what had been learned on 

• the course. The two tests used at the beginning and end of the training 
only covered certain aspects of the cpu^rse. The results were not com- 
^ municated to the teachers, nor w.ere they u$ed as a basis for alternations 

* to the training; Tffus the evaluatioh was hot integrated *in the trairiiilg^, 

which therefore lacked very £mpo37lant element, namely a deliberately 
built-in feedback system to facilitate ^ucqessive modifications and 
improvements to ^the training. • ' . 

(h) In the job situation the Health Personnel were without any kind of 
educational material such as flash cards, le&fle^ts, models, etc. ^ 
whatsoever. 

(i) The training was well directed administratively, but there was no real - 
' ' V * pedagogi^cal leadei^ship at all. ^ 

3* Changes effected , 
Introduction > » , 
In bfder to remedy the training deficiencies reviewed in the previous section^ a working -party was 
set-up. / . 

The future work was divided up for plaaning purposes into thr^ elements,, as Illustrated in 
the diagram on the next page. ^ 

The 'objective for the fhree remaining weeks of my firsty^sit being now to plot the main 
contours of a revised training programme to be tried out during the summer and early autumn. Oi 
i966. . • . ' ^ / * • 
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Modifications concerning ^ I 
the training courses. 




1 * • , 

docuynented in the form 
of ^a Teachers^ Manirai. « 


— 



The pro^ductipn of material of . 
immediate^'uiie in the/job' ^ 
situation*/ 

- leaflets, visual aids, etc. 

r directions for carrying out 
^e work , 

- factual reference material 
(e.g. regarding the side p., 
effects of different contra^' , 

^ ^ceptiVes). 




y » * » 

documented in the form 
* of A Guide tp.Family' 
Planning. 


> > 

'if 



Recommendation that the Health 
Department appoii^t a single . 
authority responsible for all"^ ; 
family plaiining training and at 
the same time invested yrith the 
pedagogical dire jti on of the, 
training^in Bandaragama. 



Taking the job description as our starting point we tried to forn^ulate the training objectives 
behavjoural terms, but this proceduife liad to be abandoned as too prodigal of tiftae. Instead the . 
training was divided up into topics, with tiie job description as background, after which behaviour In 
the job situation, training objectives, course content, jpciedia and evaluation were ditcussed in 
relatively close detail regarding each topic. The niaterial thus produced provided thejoundation of 
the Teachers' l^Hanual and the Cuide to Fayniij 'Planning^ , * 

We shall now turn to consider varioua elemelrits.in the training which were revised or 
recons^ucted, ' * . ♦ • ^ ' 

, The-framing of job descriptions ' , 

A preliminary draft of tbe job descriptions; was ready by June 1966 and was successively revised. 
The policy expressed in the following Job .description, which, together with the corresponding job 

' \ , * , ♦ 

description for PHNs> and PHIs, was approved by. the Advlpory Committee in August 1967, is that, 
pj^evlously laid down by the Health D^pairtm^ent,. making the midwife th^ key person in the programme. 

Instructions fer the guidance of Public Health Mldwives in Family Planning 



denei^al: 



li Discuss *^th the coUple - :jvh en ever possible - about Family Planning, 

2. Get in contact with the expectant mothers at the Clinics and during Home 
r Visits a?id discuss Fairiily Planning. . ^ 
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3. Meet lactating mothers/and tdlk to ih^m Airther about^what they had 
0 already been told prior to delivery. ^ ^ . * ; • 

4. Visit mothers who are practising Family Planning and encourage thfe 
^ ^ continusmce^ of the practice. * ^ 

5. Keep proper Tecords of all Family Planning activities, 
. ^ 6. Make contact with women's organizations in the area. 

Specified Duties ; . , * 

, ^ Routine Home^Visits: 

1. Estimate the number of productive couples in the area. ^ 

2. Introduce the subject when son\e problem of personal interest to the . 
family is being discussed. ^ \t - 

3. Stress the need for -spacing childreri. • . • ^ 

4. When talking to families, consideration should be given to religious 
susceptibility, economic background and social set -up of the locality -<? 
also considering the e^cational standard of the couples concerned. 

5. Whenever possible h'ave discuiersions on Family Pltoning with wife and ^ 
husband together. . ^. * 

^ ^ 6. If the circumstances are conducive talk about various methods as 

indicated below: • 

(i) Female - Oral tablets 

- Loops 

* X ' - Foam tablets 

. ^ ^ - Spermicidal pastes 

, , * . " diaphragm and jelly 

- Surgical sterilization 

(ii)Male -Condoms 
*^ - Surgical sterilization ^ 

" (iii) ' -Safe period 

- Coitus interruptus 

7. Discuss the relative value of each method.. Use educational material and 
sa^nples of contraceptives in the discussion. 

/ 8. Hand oyer leaflets on the subject. 

9. Help the coUples to choose the methods they like to practise. 

10. Supply condoips and foam tablets if required. 

11. Direct those willing to use methods requii:ing a. Doctor's examination to 
• . the nearest Clinic. Give tK'em the reference card. 

12. Clarify any false information that may have been spread regarding the 
; " different methods. of Family Planning. ^ 

Clinics {Maternal and Child Health): ' 

13. Include Family Planning in routine talks ^n various health subjects. 

14. When discussing with expectant mothers ii^ to introduce Family Planning 
if liuitable. . . ^ . . * 
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15\ Speak to the mothei^s regarding growth and development of children and 
explain how too frequent pregnancies' affect the health of children. Also 
how pregriancy and lactation affect tfie health of the mother. 

16. Whenever possible talks should be supplemented by educational material. 

17. Assist the Medical Officer With I. U.D. work if asked. 
Special Jlome Visits for Family Planning Follow-up ; 

Visit homes of those who are using family planning. methods and check i^p 
' • to see whether; ♦ ^ 

(i) they are following thp method as advised; 
o (ii) there are any sido-effects; • 
(iii) they require a change in the method;* 
(iy) they require further supplies of contraceptives. / 

19. Visit homes of those who^ are using, methods at the following stages: 

(i) as early as possible visit the mothers and see whether they are satis- 
fied with the method; j 
.(ii) for the purpose of giving new supplies of contraceptives, before the 

stock of contraceptives with them (mothers) are- over,* 
. Loop; 

(i) after being informed by the Medical Officer that a loop has been 

inserted, a visit should be pald at the first opportunity ard notes taken 
^ ' of any side-dffects; \ 

(ii) after one month; ^ 

(iii) once in three months if there are no side-effects; 

(iv) if there are side^effetts whenever visiting that particular locality for . 
rbutine work, , " : , 

^ ~ ' • Oral Tablets: 



(i) fi^st opportunity (acfcbfding to routine programme) after receiving 

inforJiiation to see for any side-effects or problems; 
(ii). before the tablets are finished; 
"(\ii) re-visit is necessary in Order to direct mothers to a clinic for 
/ ' ' further supplies or medical advice. 

Record'Keeping : * ^ - ^ - ' ^ ^ 

20. Use the following record|^ in ^connection with Family Planning work; 
(i) ante-natal card; ^ ; 

(ii) the information (form) tO' the Public Health Inspector; 

(iii) the pink card; 

(iv) reference (form) to the Doctor; ' 

^ ^v) the Receipt book; . , , 

(vi) <the Monthly Return fqrm; * 

(vii) the Yellow card; ; . 
{via) other forms, 

Poritatts with Women's Organizations; 

" 21. Keep contact with Mahila.Samities, Kantha Samities'', Rural ^Development 
Societies kr\d other Women'iS Organizations in order "^o get them to 
support the Governmental. Family Planning Programme; 

22. Enlist the cO-operation of these groups of women to organize educational 
programmes on Family'Planning, 
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Assistance from the other, staff : 

, 23. Contact the Public Health Inspector of the area in the event of the husbancj 
needing family planning advice; ^ 

24. Contact the Public Health Nurse of the area in the ev^^nt of any difficulty 
in educating the mother on family planning. 

Thus the Public Health Midwife has to, carry out family planning work along with her Maternal and 

Child Health -activities and she plays the /oUowingrores: . 

1. Role of an educator - educating couples and motivating them to use family planning 

methods, . ' / 

* ,2.' Role of a supplier - supplying contraceptives. , 

3. Assisting role , " helping couples, to get medical attention in connection with 

fatniiy planning, methods; 
• insjertioh of the loop; 

- diaphragni and jelly; • ^. 
' r sterilization. . ' / 

The jnain responsibility for the [public Health Inspector is community health education in 
order to motivate people for fainily planning. H,e would be conducting group meetings with .com-^ 
munity and otlier suitable groups.. One of his special duties would be motivating the husbahds for 
fainily planning. • ' ' ^ 

Despite the lack of precision which one might find in these job descriptions, they provided 
an excellent basic material in determining trcuning content and as directi9n8 for the conduct of the 
work in the field. * • 

\ Teaching aids and 'job aids 

In order to provide the public health with a minimum of educjational supply it was planned to^produce 
as early as possible a set of pictures illustrating the process of conception and the functioning of 
dif ferment contraceptives, ^ , . 

During 1966 drafts wei^e also made for two leaflets providing brief information on family 
planning and various contraceptives. These were distributed in 1967, an4 during the first year of 
the programme they were the only informative material available. ^ 

Mention should also be made in this connection of 'A Guicje to Family Planning'. This is a 
manual designed to give the field worker (PHMs,. PHNs ancl PHs) facts about family planning, the 
government programme and instructions for the family planning job situation. The manual serves as 
a ready referencift and reminder of what the Public Health Personnel has already learned in the 
training centre. Special importance is attached to health education and aspects such as; 

- the creation of'a need of family planning; 

- resistance to changfe; \ 

- how to overcome resistance. 

" - motivation. * , . 

* - human. relations; / . ' 

- first contact with people; ^ ' ' o ^ 
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- hints for a visit to a family; . ^ 

- hints for a talk to a group; . , ^ . , ' , 

- hints for discussions/ ' 

Theoretica l course content • 
Classroom instruction was now allotted ratjier less time, but the most important change lay in a 
re-structuring of the total content so that more scope was given to sections dealing with pre- 
conditions of behavioural change and.the practical conduct, of the work^ at the same time as more 
irrelevant and excessively complicate^cburse content was eliminated. 

As can b6 seen from Table 2, the varipus subjects were given new names designed to convey 
more clearly What they cover. .A new section entitled 'Resistance to Family Planning' was added to 
Health Education? » * ' . ^ • . , - 

^ ' The group work ^ ' " , « , ^ ^ 

Considerable changes.have been made to the group work in that the time allotted to it has been 
increased and greater emphasis has been placed on individual training. The objective oi this part of 
the training was formulated as follows: 

- to give each trainee opportunity to practise the educational aspects pf her 
'duties; * ' . . \ ^ 

- to make each trainee competent in applying the knowledge of educational 
principles and"ilsing aid material when communicating with groups or 
individuals; . 

- to help each trainee Ao acquire th;e art of introducing family planning 
without^ creating an artificial atmosphere. ^ 

The subjerct of group practice is introduced to the whole group, epiphasis being put upon the form of 

expected trainee; behaviour after such practice. The trainees are dividell into thi:e.e groups during 

ohe of the lei^sctns on health education. This part of th"6 training is then described in detail.. The 

three groups will have practical lessons at three different stages with a teacher to guide and, assist 

them. The following practical lessons are provided: 

, (i) talk to a group without aid material; ^ . , ' \* 

(ii) talk to a "group with aid material; • ■ • , 

(iii^) dialogue with a mother without aid ir/aterial; 
. '<iv) 'dialogue with, a mother with aid material. 

The groups perform separately during the first 4 lesson^. Then they meet as one single group and 

trainees perform in front of the! entire group. Lastly the cgroups meet separately during the next ^wo 

lessons, when they will be capable of more experience^ performance*. 

Field training / 

The objectiv^^of the field training is td give the trainees a thorough knowledge about how to carry out 

home visitsah connection with family planning. Each trainee haslo write field notes^regarding work 

done by Ker. Every instructor is expected to plan the day's work,and discuss same with the trainees 

before going tQthe field. The t/ainee shall be given maxiniuni re^onsibility in approaching and 

dealing with the situations while the instructor assists whenever necessary. 
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Table h Resum^^of topics Jor public health midwives 



Methods 



f o pi c s 



Class 
room 
lessohs 



OToup 
work and 
individual 



Demons - 
•tration Total 



l.# Orientation 


j(x) 




1 


2 Whv Familv^ Plarminix? ^ 




\ 




'(a)' Health reasons 
. (b) E0on6mic r.easons 






2 


a. T^e GoyerhmentalvFamily 
' ' P'lanning Prograpime 


2 






4. Contr'aceptivfe^Methods 


5 


/ ■'• 




5. Health .EduQatibn 




( 




Principles and methods .* * , 
of^health education 


3- ' 






,(b) How to dojmmiyiicate with 
a family . ' 


1 




\ • 14 « 


{c).How to communicate with 
* a group ^ ; ' 


2 






(d) Practical lessons 




8 




6. Resistance'to Family Planning 


1 " 


3 . 




7, Recordkeeping 


3 ^ 






8..;Field«training , ' 




14 


• 14 


9; The role of the health personnel in 
the Fajmily Planning Programme 


1 




\ ' 1 


10. Demolj^tratioa - mD<)lihic : 




t 


o2 2 


11. -Seminar * . !^ ' 




7 


''7 


J 2. MiscellEui.edu s ^ 

(F^ilnis on FarxUly Planning, 
extra lessons). 


3 




3 


13. Evaluation * . . * 


2 




2 


^ Tot^ 


26 

< A. 


32 


. 2 GO 



(x) The figures refer to number of lessons (one lesson = 45 minutes). 
^T^ * i I 
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In order to remedy^ the inefficiency of the field training, it was suggested at one point that 
;the number of instructors be increased so that each instructor only had one trainee to supervise. * , 
This alone would have resulted in a lOO.per cent rise^dn efficiency. 

<i 4. -Evaluation * * . 

As we have already observed, one of the leading principles in the revision of the training has been 
to test it successively so that the changes effected can be empirically implanted in both teachers 
and trainees. The training scheme devised after the f/rst .visit was tried out at th,e ;P,9int Pedro 
training centre and evaluated by means of two written tests together with systematic observations by- 
the training centre Health Educator of the functioning of the training. The experience thus gained 
led to a modified form of the training which was extended.to Bandara^madn.December 1966. 
Further changes in the training gcheme were decided on at teacher staff conferences/ vhich were 
now held regularly-every month. Thus^the training scheme describedin this part of thii riaport is 
the result of an informal and formal process of evaluation and refers to the training scheme as it 
stoodby the summer of 1967. Thus it should be noted, in keeping with the attitude previously 
adopted, that the training was not conclusively evolved but t^at it is hoped that evaluation has been 
integrated in the training system to faciUtate successive evaluation and revision as necessary^ 
' The Evaluation incorporated in the training can be said to have the following objectives: 

(a) to assess to what extent the individual trainee is reaching the stated , 
ti^aining aims; / 

(b) to assess the progress that the individual trainee is making during the' 
training; ■ , 

i {c),to determine the effectiveness of tKi teaching procedure. 

These objectives are evaluated by means of the group work,, thfi seminar and the field 
training. as well as two written tests. 

The redisposition of the group- wrrk mak^s it possible to evaluate individual trainees' 
performance and knowledge during the cout^se, at the same time a^ this evaluation can be used to a 
certain extent as a feedback to the theoretical lessons preceding the group v/ork. When a talk is 

given or a dialogue situation presented, the audience in the group are instructed to observe the 

... ^ • 

performance and evaluate it. Trainees practising giving -a talk to a group (i. e. Place of talk: Child 
Welfare Clinic in a Muslim community; Group: mother against family planning on religious grounds; 
Topic: family planning is not contrary to religion) aie directed to observe the following points: 

(a) Does she stand in front of the group correctly? 

(b) Does she use the right expressions?* 
.(c) Does she use gestures correctly? ^ 

(d) ,Has she contact with the group? 

(e) Does she hold the attention of the group? 

(fj Does she impart the facts correctly and in a relevant manner? " 

(g) Does she support h(Br words with the satisfactory use of the material? 

oo 
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(h) Does she allow the audience time to grasp facts? 

(i) Does she face the audience properly - not turn her back when using the 
material^?'* 

(j) Does she summarize at the end and finish on time? ' 
When the trainees are practising a family planning dialogue with a mother in role-playing situUions 
they are mgide to assess the. follQwing: ^ . 

(a) Dpes she attempt to win the confLdencie of the moUier? 

(b) Does she greet the mother when meeting? 

(c) Does she talk to the mother on matters of interest to the mother? 

(d) Does she introduce, the subject of Family Planning in a natural manner? 

(e) Does she use correct expressions? 

(f) Does slie use the aid material satisfactorily? ^ 

(g) Does she listen patiently allowing the mother to give her vi6ws? 

, .(h) Does she relate scientific facts to the ideas of the mother correctly? 

(i) Does she embarrass the mother? • ' * 

(j) Does she keep within the time limit? 

Each presentation is discussed on the basis of the trainees' -and the teacher's observations. One 

constantly recurring observation in the summer of 1967 was that trainees simply did not know how to 

use materials. They also had difficulty in broaching the subject, of family planning in normal 

conversation without creating an artificial atmosphere. ^ 

Written evaluation is effected with the help bf three. questionnaii^es: 

- 1 multiple-choice test; ' . . 

• 1 test with short answer items and limited-response items; ' 

- 1 general evaluation questionnaire of the type described on page 93. 

The multiple-choice test consisted originally of 35 items but was subsequently reduced to 
25. The only knuwledge that multiple-choice questions really test is that of recognition, which 
makes them of limited value when trying to determine whether objectives of other kinds have been ^ 
attained. This type of test is employed in the present context mainly for pra-^.tical reasons: it can be 
corrected rapidly and the results promptly conveyed to teachers and trainees and discussed ijri the 
course of traimng. 

To supplement the multiple-choice test another test was devised containing questions Which 
were thought to resemble the demands of the actual job situation, e. g. : 
What advice would you give a mother who uses oral pills and complains of: 

(1) Nausea 

(2) Breast tenderness « « 

. (3) Spitting blood 

Altogether the testi'icontains 16 questions which take 1 hour to answer. 

The following plan prpvides a summary of the nui)iber of questions covering different topics: 
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Topic 

2 (a) 
2 (b) «c 3 
4 

•5 (a) (b) (c) 
6 

10 



number of multiple- 
choice questions 

' 0 

5 

7 

4 

0 
25 



number of*short- 
answer questions 



2 
2 
7 
0 
0 
5 



16 



As has already been observed, a more integrated evaluation is gauged from topic 5 (d) 
(Health Education - practical>lessons), topic 8 (field training) and topic 11 (the seminar). For some 
topics the evaluation is included in the lessons, i. e. in topic 7 (record keeping) the trainee practises 
how to complete the different records and forms that are u'sed. During the lessons 'imaginary,^ <*^8es 
are used alid each trainee has to fill in the records,, and some trainees also practise in front of the 
whole group at the flip -over. 

Figure 4 shows the results of the multiple choice test given on the 1967-68 midwives* 
courses. The average pre-test standard was about 30 per cent, while the average post-test standard 
was 80 per cent. The items with the lowest post-test standard were topic 2 (b) - Why Family Plan- 
ning? Economic reasons - and topic 3 - The Governmental Family Planning Programme. 
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In the special questionnaire the trainees are given the opporti^nity to set do)yn their opinions 

of the design of the course and its shortcorriings as they see them. Tl^e results of this questionnaire 

were discussed, together with those of the tes^s, at teacher-staff conferences, and these^discussions 

have led to repeated- revisions. of \he training!/ 

^ 5. Teachers' Manual^ 

The revised training course is documented in a Teachers' Manual yhich is intended as a guide to 

teachers entrusted with implementing the family planning training |)rorramme. Objectives, lesson 

outlines, training aids and teaching hints are given in tjiat order under each topic. We can illustrate 

this by reproducing topic 2 (a): Why family pl uming? ^Health reasbns (1 lesson): - 

Objective ; to educate the trainees about the need for family planning from the point of view- 
of the health of the motherf, .the children and tlie father. 

Outline ; 1. ^ Protecting the^health oi ihe mother ;' / • o 



(a) frequent pregi]ancie j affect the health 6f the mother; 



(b) an expectant mother 
io-her womb; 

(c) breast-feeding the r 

Protecting the health oi ' 



has toj'look after herself as well as the baby growing 

ew«born places a ^train on her health, etc. 
the children: 



(a) * if mother gets pregiiant within 2-3 months after baby is born she may not 

be able to .produce e nough milk to fe^d the infant already born because of 
weakness; 

(b) breast milli is necessary for in<lnt*^ health.* Mother's milk gives tlje- 
baby a better chance to live; / 

(c) baby can be born prematurely and can become weak as a J^^sult of 
frequent pregnancies. 

3. Protecting the health of the father ; 

(a) health will be affected by overwprk in supporting ^ large family; 

(b) if baby is weak and sick father cannot rest. 

Training aids ; Flannelgraph - words and terms in colour. 
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BANDARAGAMA CENTRE 

QUESTIONNAIRJ] TO BE ANSWERED AT THE BEGINNING AND AT THE 
END OP THE TRAII4ING * 



/ 



1 



Which factors play a considerable role when a family decides to practice Family Planning? 

bo yoii think 'that a family Urould change its attitude to Family Planning in case any-of the 
above factors mentioned by you in>reply to question 1 would change? * 

To what religion do you belong? \ , . . ^ 

Do.you think that your religion is in favouj: of Faniijy Planning or not? 

(a) explain why not in favour: 1- * 

, (b) explain why in favour: *. * ' . • . 

0 When paying Family Plannfiig home v;isits to a family, there are advantages and 
disao'vantages- ^ 

(a) what are the advantages? > 

(b) what 'are the disadvantages? " ^ " 

Please mention what Faniily Planing methods could be issued- , • \ 

(a) in the field? . ^ ' , 

(b) in the clinic? - , « o ^ 

When do you think a family is most favourable to Family Planning? 

^ — 

^ (^)obefore chil(Jbirth? * - . • . ^» 

lb) after childbirth? * • ^ j? * - 

• »% ' 

^If 'the family is indifferent lo* Family ]?Janning, what can i ■ 'reason be? 

How can you overcome the reasons mentioned^'in reply to question 7? ^ 

What are the points you would consider in teaching a trainee in the field? 

Mention the ways fiow Pujslic Health personnel could isaist lilk Family Planning Programme 
in a rural area^ * , ^ * - 
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GUIDELINES FOR I.B.R.D. •S EDUCATION SECTOR REVIEWS 
AND EDUCATION ij PRE-INVESTMENT STUDY PROGRAMS 



by M. Hultin and C. van Dijk 



1. OBJECTIVES Of THE WORLD BANK GROUP'S FINANCING 
IN EDUCATION 

1. 1 The World Bank Group's objective of financing education in developing countries is the , 
promotion of economic development. The Bank is, however, fully aware of the social, environ- 
mental and cultural objectives to be served by the education systems, and a growing knowledge 
of the effect of these factors on development has led to the formulation of broader Bank objectives. 
Those broader objectives have been reflected in the Bank's Education Sector Working Paper of 
September 1971. The Bank, however, continues to focus its operations on education projects which 
are oriented towards economic development. 

1. 2 Increased emphasis is being placed on the qualitative aspects of education. Existing sys- 
tems of education and training are not effectively contributing to development because of deficien- 
cies in orientation, content and technology. The Bank is giving particular attention to projects 
which would lead to improvement, modernization and economic relevance of the education and train- 
ing sectors or sub-sectors. 

1 3 The serious shortage of financial and other resources ma> defeat these objectives unless 
the internal efficiency of education systems is improved greatly. A major objective of Bank financ- 
ing should, therefore, be the promotion of efficiency so that improved* outputs will be obtained with 
lower inputs of scarce resources. To attain this objective new structures, techniques and changed 
input combinations may be required. 



1/ The word 'Education' in the title (and in the text if not stated otherwise) 
covers both what is traditionally defined as 'Education' and what is defined 
as 'Training'. The word 'system' is used in the widest context and includes 
'formal' as well as 'non-formal' activities. 
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1.^4 In short, the Bank's objectives are: first, qualitative improvement of the education and 
training systems to meet economic and other priority development needs, second, increased effi- 
ciency to reduce cost levels to the limits of available resources, third, quantitative expansion of 
facilities where such expansion is justified by manpower demands, fourth, broadening the base of 
human resource development by increasing opportunities for basic education and skill formation 
outside the formai school system* 

2. OBJECMVES OF EDUCATION SECTOR REVIEWS 
2. 1 The objectives of education sector reviews are to: 

(a) Analyze the education system of a country, and" its capacity to promote 
economic, social, environmental and cultural development efficiently; 

(b) Suggest, whenever necessary and possible, ways and strategies to 
improve, expand or reduce the education and training system so that it 
will achieve aims which are relevant both to the society and to overall 
development of the country; and, in close conjunction with the proposals 
in(b); 



0 



(c) Identify priority investments in education, and 

(d) Prepare programs of pre-investment studies necessary for development 
of the education sector. 

2 2 An education sector review should go beyond the objective of World Bank financing as des- 
cribed in Section 1, and should deal with desirable social, environmental and cultural objectives of 
education and training, since no comprehensive education sector review can be made without con- 
sidering these aspects. The review should, of course, primarily serve the country in which it is 
undertaken, and this country will undoubtedl> also be interested in education objectives other than 
those directly promoting economic development. The review should, furthermore, serve agencies 
other than the Bank Group which might have somewhat differing education policies* The main em-y 
phasis should be, however, towards pre-investment studies and projects for economic progress, 
with consideration to underprivileged areas and population groups* 

2 3 The main-purposes of these guidelines are to: (a) describe tlie necessary analysis preced- 
ing the identification of education investment projects and of pre-investment studies, (b) describe 
various aspects of education pre-investnient studies, and (c) describe the execution of education 
sector reviews. 



3. STRATEGY OF EDUCATION SECTOR REVIEWS 

3 1 Until recently, international and bilateral agencies* activities in education have been mainly 
concerned with assisting the developing countries in expanding their education systems largely within 
existing frameworks. Although those frameworks have been questioned from time to time, few 
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serious attempts were made to review from scratch aims, contents/ structure, technology and 
financing of the education systems. One reason v/hy a time-consuming overall review of existing 
systems was seldom undertaken was the urgency of expanding education within the shortest possible 
time to meet a desperate shortage of manpower in the 1950»8 and 1960»s. Another reason was a 
lack of awareness of the system^s low external productivity and internal efficiency and of its cost- 
liness in relation %o available financial and manpower resources. 

3 2 Further expansion of existing education systems would in many cases lead to a qualitative 
deterioration and excessive education expenditures. There is, therefore, a consensus among majoi 
developing countries and international and bilateral education agencies that changes aimed at increas- 
ing external productivity and internal efficiency and decreasing production costs per graduate are 

essential prerequisites to further expan&ion. Necessary changes can hardly be achieved, however, 

'J 

within existing frameworks. New approaches are necessary and old concepts must be systematically 
questioned, The traditional tripartition of the system into primary, secondary and higher education 
may no longer be feasible. Boundaries between formal and non-formal education, as well as bet- 
ween education and training may be artificial and may be abandoned. The formal education of a 
child in the 7-12 years age bracket should not a priori be considered as more important than the 
training of the adult. A good pre -school environment and suitable nutrition are essential for the 
child's intellectual development and appropriate parents* education may be a method to achieve this. 
The concept of lifelong learning should not be overlooked and the importance of informal training for 
those outside the formal system must be emphasized. Adnninifitrative boundaries between mass 
media and education should not be allowed to prevent a full and integrated use of such media for 
learning purposes whenever feasible. Existing concepts of utilization and composition of staff, phy- 
sical iacilities and tne academic year snould not conceal the fact that those concepts are babed on 
traditions of an old society without access to today's technology and means of communication and 
transportation. ^ ^ 

3. 3 It would be unrealistic to believe that the short time generally available for an education 
sector review executed by expatriates (compare Para, 3. 5) would permit the creation of an entirely 
Ttew education system, and"there- are education elements-which shoul^'sv.rvive^^in' ever;^"System. 
Nevertheless, a searching inquir>^ into all parts of thft existing system, together with evaluation of 
alternative structures and technologies, might be a useful and necessary first step to bring about 
greater willingness of countries to experiment and innovate. 

3.4 The reviewers should, thereifore, look at a country's education and training system in the 
widest sense and evaluate the role of the system in promoting overall development. It should be 
realized that each part of the system should comprise a building block of a complete education pyra- 
mid. The purpose of each block is chiefly to prepare the student for a place in the society, rather 
than to prepare him for a place in the next stage of the education system.j But a free flow of students 
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within the system should also be possible. The relation and interaction between thelabui nictrkot 
and the education system should be studied in depth. Much attention should be given to .he employ- 
ability of school leavers and .the establishment^ of a school-leaver tracer system naght be desirable, 
3, 5 Since an education sector review mus* fully reflect the country's needs and aspirations and 
lead to effective implementation (which must include provisions for a continuous debate, discussion 
*and review pf the kind of issues with whi-h the survey will be dealing), the country's own educators 
and other specialists should ideally be responsible for and execute the review. The role of outside 
experts would be to provide technical know-how as well as appropriate information of recent develop- 
ment of education in other areas of the world for comparative purposes. An education sector review 
of the iy^ and depth described in these guidelines should be undertaken by teams of expatriates 
under special circumstances. Such circumstances might be the need to identify education priorities 
and projects within the context of a multilateral or bilateral development program in which time and 
manpower constraints prevent a survey fully undertaken b^ nationals. In other cases the country 
may consider an evaluation of the education system by an external organization an important asset 
to arrive at an objective and politically indeptndent proposal. The country's own educators and 
other specialists should, however, alw^ays be involved in the work of the survey team to the extent 
possible and consistent 'vith the overall objectives of the survey. 

3,6 An education review should i^tart with the aims and objectives of the countr:y's development 
^^^Q^ts. What kind of society is expected in political, economic, social, environmental and cultural 
terms'^ What is the planned economic growth? Are the expectations realistic? What would and 
should be the roles of the major social and ethnical groups in the future? What would be the role of 
the woman*^ How might and should agriculture, industry and trade develop? Have the expected or 
planned develoi^ments been conceived in operational terms? Is it possible to define the demands on 
the education system which the prospective overall development oJ the society would cause? 
3, 7 The review of the country's overall development shall sho v the way to and guide the discus- 
sions of the aims and objectives of the education and training s.ystem, ^orma. as well as non-^formai 
and its external productivity. Are the proclaimed educational aims and objectives relevant to the 
development efiorts? If the aims and-objectives^are relevant, are they also-achieved - i. e. is the ^ 
external productivity satisfactory? How do we know that they are achieved? Aims and objectives 
are often vaguely defined and seldom described in such operational terms as would perniit quantita- 
tive evaluation. They ma> have little relevance to development and a difficult but necessary task for 
a study teaXti would be to define alternative aima and objectiveTin such cases. Only in the light of 
such redefinition is it possible to proceed with a fruitful analysis of the system. The oiscussion on 
education aims and objectives must be related to such factors as the country's demography, the 
rural and urban development and migration, tribal and religious divisions, underprivileged areas 
and population groups, possible uneven distribution of the country's assets with pockets o/ poverty, 
possible areas of cultural isolation, employhient^ semi-emplo>ment and unemployment, bureaucratic 

100 

98 



1. B.R. D. 's education sector reviews and education pre-in vestment 
study programs 



characteristics, pattern of leadership roles, etc. A discussion on objectives must dwell on 'internal^ 
factors ^n addition to the 'external* factors mentioned above. Such factors are the cognitive, attitu- 
dinal and behavioral achievements which should be expected by the participants in the learning sys- 
tem Is literacy achieved as scheduled? The external and internal factors have All important impli- 
cations for education and training requirements which must be considered. 

3 8 An education system's demands for facilities and staff depend on content, size, structure, 
technology and other factors discussed below, and a final costing of a system cannot be made until 
all those factors are known. However, available resources have a decisive influence on the shape 
of an education system, and tentative resource parameters should be established at the beginning of 
the study and an awareness of costs and staff supply is necessary throughout the study. The inter- 
action between the resources^plied and the education development attained should be examined at 
each stage of the study. 

^- ^ "^he second step in the review comprises an analysis in depth of the formal and non-formal 
education and training content and structure . Such review must be comprehensive and cover the 
whole system and the way its different parts fit together. How does the content meet the objectives 
as they are defined or should be defined? What is the correlation between the content and the .de- 
mands of society and labor market? Is the education and training sufficiently practically oriented? 
Can the graduates from universities and vocational institutions enter the labor market without unne- 
cessary in-service training? Are possible problems of environment, over-population, tribalism, 
religious diversions, land distribution, law enforcement, tax collection and other factors mentioned 
in Para. 3. 6 reflected in the education content? How is the 'educational profile' of the society com- 
pared with the school's educational profile? How do content and structure interact? Does the 
structure give sufficient opportunities to rural students, to women, to illiterate adults? Are the 
rural areas filled with incomplete primary schools which force the 9-iO year-olds to new scnools m 
other villages if they want education beyond grades 2-3? Are non-formal education and training 
given the proper role in the structure? In particular, is training within industry and other enter- 
prises supported by the government to achieve an optimal use of all resources? Does the structure 
provide for the use of staff and facilities during evenings, vacations? Does the structure create a ~ 
burdensome boarding system? How significant are local movements such as the Harambee schools 
in Kenya*? Are there non- economic explanations for important aspects of the education structure 
and content (preference for boarding schools, etc. )? In those cases where structure and content 
are irrelevant to the objectives of the education system, the ideal sector* review should discuss 
alternative contents and structures which would better meet these objectives. 

3.10 The third step is an analysis of the technology of the education system. The word 'techno- 
logy' is used in its widest sense and includes learning methods, management and staffing. A discus- 
sion of learning methods would cover team and micro-teaching, programmed learning, pupils' 
involvement in practical activities, cori^espondence education, radio and TV education, traditional 
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classroom instruction, achievement evaluations, examinations and use o( objective tests. A discus- 
sion on management would include centralized versus decentralized systems, the relation and coop- 
eration between Ministries (Planning, Finance, Education, Labor, Rural Development, Health, etc.), 
between levels (Ministry of Education, universities, regional authorities, etc. ), It would, furtlier- 
more, include information collection and dissemination, decision procedures and implementation, 
including possible participation of teachers and students, performance evaluation and supervision, 
use of systems analysis and modern data processing, program budgeting, accounting and auditing 
procedures, etc. The sufficiency and reliability of the education statistics (including the financial 
sections) should be given particular attention. Does the administrative setup support a system of 
'rolling education reforms'? In staffing, the diversification of the staff, staff training and staff uti- 
lization are of particular interest: Is professional staff used for sub-professional tasks? Are there 
social or cultural reasons for the adoption or rejection of specific technologies in management, 
teaching, staffing? This question is particularly relevant when it comes to the introduction of new 
^ media. Also in this context, a discussion of alternative technologies to improve the system is 
important. 

3 11 The fourth step in the analysis should deal with programming and physical plai(ining of the 
education system. To what extent have education and training objectives, structure, curricula, tech- 
nology, population distribution, urbanization, transportation me.ans, overall econom^i' and financial 
constraints been considered in the physical planning, location and construction of schools and of other 
education and training institutions'' Is there a fair distribution of educational opportunities between 
urban and rural areas and between the different socio-economic strata and groups of the population? 
Does the system meet the quantitative demands of society and labor market? How do available edu- 
cation plans and projected uniolmuntb and development of staff and physical facilities relate to eco- 
nomic plans, population estimates and future manpower needs? Is there already an overproduction 
of graduates in some education and training sector or a possible future oversupply? Is tiie current 
and future supply and distribution of staff, physical facilities aiid learning materials appropriate in 
quality and sufficient in quantity? la the utilization of staff space and materials optimal? Are teach- 
ing-staff and-school facilities beir g used for any -relevant non- educational activities? Are-other-per- 
sonnel than teachers and other facilities than schools used for learning activities in countries with a 
shortage of conventional education opportunities? Are school designs systematically studied? Has 
the 'open classroom' concept been discussed? Is there a local production of leaiaing materials, 
including textbooks? If not, should such production be stimulated? Are there any areas in program- 
ming and planning in which the relationship between politicians, administrators, planners .and tlie 
school staff are particularly difficult? 

3.12 While awareness of costs is essential during the consideration of the previous four steps, . 
the fifth and important step of the re\iew would be an analysis of the costs and the fin a ncing of Jne 
education system and its internal efficiency , the cost per unit of production. Thi« analysis should 
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relate the costs of education to other expenditures of the society. Education represents both invest- 



ment and consumption and in both respects should be exposed to intra- as well as to inter-sectorial 
priority reviews. The allocation of funds between education levels should be analyzed. The appro- 
priateness of the higher education budget should be particularly scrutinized, as higher education in 
many countries receives a larger part of scarce education resources than can be justified. Staff 
salaries should be re\iewed and the reasons for possible rigidities and anomalies explored. 
Teacher^s salaries form 80-90 per cent of recurrent costs in education in many countries and ways 
to reduce that percentage might be uiscussed. Administrative practices in hiring staff which some- 
times inrrease the education costs should also be studied. Funds for school maintenance and pur- 
chase of book5 and materials are often insufficient and nov only the need but also the effective demand 
for such services and materials must therefore be explored. The review should discuss existing and 
other possible sources of Education financing for capital and recurrent expenditures, such as self- 
help schem3s, loan funds to students, in-plant training with special taxation of enterprises, etc. 
The cost analysis should give cost ranges, sensitivity to changes in inputs and, as far as possible, 
alternative costing solutions for improvements of the education system which may have been dis- 
cussed previously in the review. The use of rate of return analysis for different types of education 
and train-'ng has been discussed in the Bank but is not recommended for use in this context. The 
^ ^ financial analysis should project the probable growth in capita] and recurrent educational expenditure 
arising from all aspects of the educational development program and compare this growth with esti- 
mated increases in GNP, government revenue and general expenditure. The discussion on internal 
efficiency would not only deal with dropout rates, the repeater problem and the reconstruction of the 
true student cohort but also explore the students* achievements at the different levels of the educa- 
tion ladder and their terminal performance. Such exploration should include attitudes and behaviors 
as well as cognitive skills and knowledges to the extent piossible. 

3.13 The above five stages of the review should lead to a sixth and final step, which is the pro- 
posal for an action progran r:. If a sector mission has done a thorough job of analyzing the education 
systeiTi and the direction in which it is moving, ii "^ould be possible to examine alternative ways of 

"remedying weakne¥ses~m^ system, to propo::,e a strategy for future development, and^to recom- 
mend specific policies for all levels of education. The program should be phased to the country's 
overall development program, have a well-defined time pattern, include costing and give priorities. 

.Financial and staffing constraints should be indicated. The action program, should be formulated and 
organized to permit the full use of available local talents, and the ultimate goals of the program 
should be to create an education system which, while promoting economic and social development, 
can be fully supported and-furtKer developed by the country's own manpower and capital resouixes. 



be proposed when absolutely necessary for the program execution and the institution building. When- 
ever feasible, alternative ways of achieving the goals should be stated. 



Technical assistance should not be proposed as an qasy way out of a staffing-problem. Jt should only 
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3. 14 The t)rogram should include a list of identified priority project s, which might be financed by 
the World Bank Group or any other agency. Annex! contains examples of possible projects which 
may cover any part of the education and training sector. It is important that such .projects be well 
justified and limited to what is necessary to achieve the development objectives. 

3. 15 The action program should also define in detail the work that would have to be done and studies 
to be undertaken prior to the implementation of projects. The sector review would thus contain a^ 
program of pre-investment studies. Pre-investment studies would be an aid in the design and pre- 
paration of projects, and in many cases would constitute a prerequisite to the implementation of a- 
project. They would form an important tool in reforming the education systems by recommending 
means to increase external productivity, quality and internal efficiency. These means would then be 
reflected in the actual projects A list of possible types of pre-investment studies, is given in 
Annex II. , 



4. CRITERIA FOR PRE-hWESTMENT STUDY PROGRAMS 
IN EDUCATION 

4. 1 The identification of pre-investment studies is a delicate task because errors or omissions 
could have serious consequences for the overall development as well as for the execution of projects. 
When proposed education projects would depend on previous or current studies, e. g. , manpower, 
curriculum research, learning materials production, new media, etc. , the quality and coverage of 
those studies should be carefully examined. When projects would depend on proposed new studies, 
it must be assured that all relevant xopics will be covered. This coverage is important, as pre- 
investment studies may constitute starting points for a continuing development in curricula, educa- 
tion cost analysis, school planning, etc. The task of identification of a pre-investment study should 
include estimates of its requirementb for technical assistance and counterpart training, but not any 
possible soux'ces of such technical assistance or training. 

4. 2 The education and training system in many developing countries is in such a shape that the 
temptation-exists to-propose a^engthy catalogue of studies-covering the majority a^topics hsted in 
Annex II. Implementation of auch a lengthy catalogue would in most cases be beyond most countries* 
capacity to administer and to absorb, and would delay development as well as investment. It is 
important, therefore, that the sector mission should exercise restraint in proposing pre-investment 
studies. In each case i* should be clearly indicated whether action on a specific project would be 
dependent on execution of a proposed pre-investment study. 

4.3 In a few cases education sector knowledge on a particular country may be so good that a 
pre-investment study program can be produced as a desk study rather than through a field visit. In 
such_£ases^ it should be remembered that the Bank Gro.up»s approach to education has only recently 
broadened. Previous Bank^^nd Unesao, missions might have reviewed the education systems from a 
relatively narrow angle. J-^^J- 
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4.4 Pre-investment studies may be recomciended with or without Bank Group project financing. 
If project financing is foreseen, the financing otiiiSapre-investnTient stud)* might be achieved in vaiioub 
wa>s, either by the Bank Group or by another agency. Whatever arrangement is recommended, the 
decisive factors should be the proposed study^s contribution to the accelerating and iniproving tlie 
preparation of projects and to the execution of investments* 

4. 5 An education sector review mission would normally have neither the time nor the expert 
knowledge to design the proposed pre-investment studies in detail. 'Although the scope of such 
studies must be defined in broad terms, proposals should be sufficiently precise to enable time and 
cost estimates to be made. The objectives of each study shguld be defined clearly. Most education 
pre-investment studies are expected to initiate action and induce change. Concrete programs for 
implementation of proposed reforms must, therefore, constitute an important part of the studies, 
and this should be reflected in the stated objectives. If several studies are proposed^, they should 
be ranked in order of priority or arranged in a logical time sequence. 

5, IMPLEMENTATION OF EDUCATION SECTOR REVIEWS' 

5. 1 An education sector review will include sector aneilysis, pre-investment study program and 
project identification. Project identification must not be relegated to the third place of importance, 
it is intended that the projects thus identified will be better conceived and, by the time they are 
financed, fully backed by the requisite pre-investment studies. 

5. 2 x\n education sector review as described in previous paragraphs will apply a couiprehensive 
approach to a country *s total education and training system. The analysis should show changes and 
trends in the system and would lead to an action program. Such an approach is more demanding 
than a traditiotial education survey and niay require bigger missions and longer duration. Proper 
preparation and implementation of a sector review will, however, keep staff requirements and Xiiue 
within reasonable limits. Although a new way pf treatment of available data would be introduced, 
collection of more information than has previously been require^ by education missions would not 

mecessaTrily brTrecessary^. " ^: - - ^ _ - „ . _ 

5. 3 For missions composed of expatriates (conipara Para. 3. 5), a saving in time and manpower 
for data collection should, in fact, be possible as nationcil authorities and international organizations 
working in the country, such as UNDP and Unesco country representatives, might be requested to 
assemble and provide specific information before the start of the review. A review of already 
existing studies and programs of recent date covering one or more education sub-sectors might also 
save mission time and manpower and provide useful inforriiation for the mission's own proposal. 
Such review may be undertaken in the Bank's headquarters before the ndssion departure by a mission 
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Bank Group and other agencies should create a unifoVni approach to education and training and 
increase the exchange of relevant information, thereby reducing the workload of a sector mission. 
When possible, deployment 'of mission members in the field should be phased so as to maximize the 
continuous flow of information. 

5-4 As mentioned previously, an education sector review mission composed of expjatriates 
would not be expected to conceive an entirel> new education system. Nor would it be able to produce 
an education development plan during the time at its disposal. Hovtrever, the action program would 
contain concrete proposals which would either support an acceptable existing development plan and • 
assist in its execution, or suggest alternative strategies for future development and ways to imple^ 
ment those strategies. It phould in both cases speed up the regular planning process within the 
country and aid national planning agencies. 



6. COMPOSITION AND SCHEDULING OF EDUCATION SECTOR 
REVIEW MISSIONS 

6. 1 The new broad approach to education and training should be reflected in the composition of 
education sector missions . Certain areas of study relate to the whole sector and require dif-. . 
ferent typec of expertise. They are, however, interrelated and should be handled with an integrated 
approach. These are: 

(a) The social, environmental and cultural factors of the country; 

(b) The demography of the country; 

(c) The external productivity, the economic factors, and the constraints of 
the sector; 

(d) The internal efficiencv of the sector; / ^ 

(e) The planning of the sector. 

In addition, the evaluation of the following three subject areas requires special knowledge: 

(f) General education - i. e. , those elements of education which are essential 
to the defined level of training regardless of the environment; 

(g) Urban development - i. e. , all those elementii in addit.on to the general 
education which are peculiar to the urban ehv/ronment and living; 

(h) Rural development - i.e., all those elements in addition to the general 
education which are peculiar to the rural environment and living. 

How these eight areas might be covered by expert staff would be determined in each particular case. 

A typical sector review mission might include a sociologist/ deniographer, an economist/human 

resources expert, \ general educator/ planner, a specialist in industry and commerce, and an 

agriculturist. Other specialists, including experts on new learning media, materials production, 

adult educauuii, nealth education, etc. would be provided as necessary. However, the aim should 
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be to ensure that' eacji niission wuifld be full> capable of dfes'essing the formal and.non-fornial edu- 
cation and training system against a background of the country's overall development and its demands 
on other sectors. 

6. 2 The actual si^e and coniposition of a full sector nfiission and the time in the field would be » 
decided in^each s{)ecific casd, although a team of 6-8 members - expatriates and/ or nationals - 
should be appropriate in most countries. It niay be necessary to spend 1-3 months in the lield and 
3-5 months for report writing. ' ' • 
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SOME POSSIBLE AREAS FOR-BANK GROUP PROJECTS 



1. ^ Research and experimentation projects with new curricula and learning methods. 

2. Assistance in long-range educational planning, curricula design, school management and 
staff training. ^ ^ 

3. Development and" introduction of instructional television, school radio, progrE^nmed 
learning, correspondence education and other technologies. 

4. Development and production of learning materials including textbooks. 

5. Establishment of education development centers. 

6. Pilot programs and schools in primary education. 

7. Functional literacy programs. 

8. Construction and equipment of s^econdary.and post-secondary institutions in general, 
industrial, agricultural, commercial, business management, social and para -medical 

' education and training. 

9. Construction and equipment of university faculties of science, engineering, agriculture, 
veterinary medicine, education, business and busine&s management, and social science, 
and other clearly development oriented subjects. . ' 

L^-! , - -'^5.^^"^^^, assistance for the implementation of proposed subjects. 



IPS 
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SOME POSSIBLE AREAS FOR PRE-INVESTMENT STUDIES 



Human Resources Development * 
1. 1 Manpower studies. 

1. 2 Rate of return studies. 

1.3 Employment of school leavers, # , . ' - * 

1.4 Tracer system of school leavers. 

Education Aims, Contents and Structure * 
2. 1 Stqdy on education structure, f 

2. 2 Development of curricula in specific areas or levels of ejJucation and training, 

practical subjects and courses, AirsQ or urban orientation, ^literacy programs, adult 
education. ^ ^ ^ ' Z^. 

2. 3 Study of performance and of examination and promotion systems, evaluation methods. 

Education Meth ods and Research 
» : 

3. 1 Development of plans for use of neV/ media or new technologies (radio, TV, 
programmed learning, team teaching, correspondence education). 

3. 2 Development of. education development centers. ^ 
-3. 3 Research on education contents and learning methods. 

Financing of the Education System 

4. 1 Cost analysis (the whole or a part of the system). 

4.2 Costxpffectiveness. * ^ ' 

4. 3 Study of education financing (the whole or a part of the system). ^ / . 
Physical Planning and Programming of the Education System ' 

5. 1 Survey of existing school facilities. 

5. ^2 Study of school building design, building materials. ' / 
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5. 3 Physical planning and location of new schools or of other education and training 

institutions and courses for the whole system or for specific areas considering: 

(a) Required enrolmment to meet manpower demands, 

(b) Structure, 

(c) Curricula, 

(d) Learning methods, 

(e) Staff supply, 

(f) Population distribution, with due consideration to underprivileged 
areas and groups, 

(g) Tribal areas, 

(h) Urbanization rates, 

(i) Transportation means, 
(j) Boarding frequency, 

(k) Economic school size (Pre-primary level, primary level, secondary 
level, higher education level, adult training, literacy). 
5.4 Development of plans for local production of learning materials (including textbooks). 
Administration and Staffing of the Education Systems 

6, 1 Educational management (central, regional, local, information collection and 
dissemination, education statistics, decision procedures and implementation, 
evaluation, supervision, use of systems analysis, network analysis, data pzocossing, 
planning, program budgeting, accounting, auditing). 

6. 2 Development of staff and teacher training programs - supply, demand. 
6. 3 Development of counterpart programs. ^ 
6,4 Development of technical assistance programs. ^ 

6. 5 Advisor}' committees for education, student participation in administration, exchange 
of information between representatives of industry^ business, agriculture, health and 
the education system. 
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Annex III 



^ CHECK LIST FOR EDUCATION SECTOR REVIEWS 



1. An education sector review must rest upon a large body of information on the education 
system, its relation to the society (external productivity) and its internal efficiency. Such infor- 
mation should not be a static description but a dynamic analysis showing flows, changes and trends 
with great consideration to the time factor. The possible opportunities to compare actual achieve- 
ments with goals and targets as established in development plans should be used. The implenten- 
tation of educational anges should be observed and studies of education programs and reforms 
which have failed may be revealing. The interactions between the labor market and the education 
system, and between the overall economy of the country and the education expenditures are parti- 
cularly important, 

2. The information should comprise: 

2. 1 Human Resources 
The country^s present resources of trained manpower; employment statistics by economic 

sector and by occupation and educational level, use of expatriate high-level manpower. Utilization 
of existing manpower resources: unemployir^ent, underemployment and emigration or immigration 
of trained manpower. Employment of school leavers. Evidence of current shortages b^' econundc 
sector and by occupation. \Va^ i differentials among occupations. The educational profile of the 
labor market. Social attitudes toward employment and use of female manpower. Recent trends in 
employment, in migration of manpower between rural and urban areas, in emigration. Comparison 
between actual employments and those originally projected for the period. Prospective require- 
ments of manpower by sector of the economy, by occupation and by area, if possible during the next 
ten years. Requirements resulting from projected economic expansion, from repatrifition of 
expatriate staff and from normal replacement, 

2.2 The Education and Training System 
(a) Current structure of the educational system. Recent changes in structure, in length of 

schooling, ih entrance age, in school year, in school and class sizes. Prospective 
reforms. -A ^ ^ 
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Enrolmen ts and outp uts in all types of universities, scnools and training ct?nterb, d> 
grade and sex and age, current and, if available, for the past five years. The educa- 
cational pyramid. Data on attrition and repetition rates c d levels at which they occur. 
Functional literacy programs and other types of adult training. Numbers in training 
and education overseas. Significant contributions to manpower training made by other 
ministries, industry, the military forces, organized apprenticeship schemes, com- 
munity development schemes, etc. in various kinds of formal and non-forxnal activities. 
Recent changes in composition of student body, in racial, tribal and social distribution, 
in ratio of male: female enrolment, in urban:rural enrolment, in technicahacademic 
enrolment, in science :arts enrolment, in^attrition and repetition rates, ^in number of 
applicants, number of places, in index of opportunities between different levels of 
education and between different geographical areas, in ratio day students:boarders, 
ratio private:public school enrolment, Prtf'spective enrolments ana outputs m existing 
and projected institutions, if possible during the rlBxt ten years. Correlation between 
prospective outputs and prospective manpower requirements as described in Para. 2. 1, 
including an estimate of school and university leavers not entering the labor market. 
Desirable outputs, to meet requirements. 

M Analysis of educational and training objectives, curricula and syllabi . Instruction hours 
per week and year. Languages of instruction. Promotion and examination systems and 
results. Educational achievement at different levels. Curriculum research and experi- 
mentation. Career 6 guidance. Aptitude tests. Correlation between education program 
and the manpow§r program; liaison between the vocational education program and 
industry and other employers. Correlation between education programs and social, 
environmental and cultural demands of the society. Recent trends and changes in 
objectives, curricula, syllabi and achievements of students. Planned reforms and 
desirable reforms. 

(d) Educational Technology 

(i) Learning' methods su(± as classroom instruction, team teaching, school 
radio and ITV, programmed learning, computer assisted instruction, 
subject integration, correspondence education. Recent trends and 
reception of new methods. 

(ii) Educational management and levels of education authority, related to 
one another (local, regional, national, private). Information collection 
and dissemination, quality of education statistics, decision making, 
implementation and supervision of decisions. Use of systems and net- 
work analysis, data processing and up-to-date accounting methods. 
Participation of laymen, students and staff in the administration; for 
counselling and for decision making. Student unrest and its causes. 
Recent changes in the management systems. Has the process of curri- 
culum reform been built into tlie system? 
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(iii) Staffing: training, utili/.ation and qualification ol staff. Staffing 
structure. Turnover. Salaries compared with free market. Social 
standing of the teaching profession. Class contact hours per week for 
,teachers. Studentrteacher ratio. Recent trends in staff qualification, 
in nationalization of staff, in studentrteacher ratio. Use of technicians 
to assist administrative and teaching staff. 

(iv) Planned changes and desirable changes In technology, 
(e) Educational Planning 

(i) The planning and its relation to the overall development planning. 

Objectives, methods and organization. To what extent have manpower 
demand, educational and training objectives, structure, curricula, 
technology, population distribution, urban and rural development, trans- 
portation means, overall economy and financial and staffing constraints 
been reflected in the planning? Recent changes in planning methods. 
Planned changes and desirable changes in planning, objectives, methods 
and organization. 

(ii) The physical planning, location and programming of education and 

training ^facilities. Regional and provincial distribution, urban, slum 
and rural schools. Design, construction, production, standard and 
maintenance of physical facilities and learning materials. Supply, 
demand, utilization of physical facilities and learning materials. School 
building research. Planning methods. Recent changes in planning 
methods and school design. Planned changes. Planned development of 
educational and training facilities. Desirable development of educational 
and training facilities during the next ten years, considering planned 
education reformSv^and/ or expansions, indicating proposed changes in 
structure and demand for physical facilities. Desirable production of 
learning materials. 

(iii) Staff development. Current supply and utilization of administrative and 
teaching staff. Projected demand, supply, training and nationalization 
schemes. Desirable supply, training and nationalization schemes to 
meet expected demand for staff. 

2. 3 Education Financing ^ 

Present , past, and estimated future requirements of the educational and training system 
corresponding as closely as possible to categories and annual breakdown as defined in preceding 
paragraphs. The analysis would include capital costs, recurrent costs and sources of funds, present 
and future. The percentage of total public expenditure represented by public expenditure for educa- 
tion, as well as the percentage of GDP devoted to education should be given. This section of the 
report is very important and should include a solid cost analysis and^ if possible, discuss the 
internal efficiency of the system in terms of cost per student and cost per graduate. Staff costs and 
effective demand for learning materials should be explored. The maintenance policy should be 
discussed. Past cost trends should be examined. Prospective financial requirements sliould be 
discussed^ C9nsidering alternative education developments as ma^ have been previously described, 
and with alternative financing methods wherever possible. Cost estimates should give ranges and 
inform on the sensitivity to changes in inputs. Foreign assistance in the past as well as expected in 
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the future should be reported if possible. As it was emphasized in Para. 3.11, the probable eco- 
nomical and financial constraints to a future development of the education system should be explored 
in depth. Such constraints are essential and decide in the final analysis on possible expansions and 
improvements of the system. In most cases the Bank would possess a considerable amount of 
information on a country's economic and financial situation which could be made available to an 
education sector mission and assist it in its evaluation of the education system. 
3- The review and analysis in-depth of the education system \yould lead either to an endorse- 

ment of existing education development plans or to a proposal for an alternative strategy for educa- 
tion development. In both cases th'e review would suggest priority projects and a pre-investment 
study program . 

4. Project identification procedures and education projects are discussed elsewhere but the 
sector mission should feel free to suggest projects in any area of education and training as suggested 
in Annex I. 

5. Possible areas for pre-investment studies are listed in Annex II. The pre-investment 
study program would also list studies which may be either project related or sector related. They 
should in every case be well defined and reasonable in scope and size. The proposal should include 
an inventory and appraisal of ci^rrent and completed studies of the education sector, giving data on 
the status of each. For each proposed new study, objectives, scope, timing, staffing and cost 
estimates should be provided. When scheduling a pre-investment study considerati^?Vi should be 
paid to the scheduling of the inve$tment. 
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SWEDISH CRITERIA FOR PROGRAMME AND PROJECT SUPPORT 



by The Swedish International Development Authority 



BACKGROUND 

The guiding principle of Swedish international development assistance is the recognition that planning, 
implemention, and evaluation of development programmes and projects is primarily th.e concern and 
responsibility of the developing country itself. Swedish assistance is country oriented inasmuch as 
it proceeds from the needs, objectives and priorities of thb recipient country. Swedish contributions 
are based on specific requests for development assistance submitted through the established official 
channel. Such requests should ideally be accompanied by /descriptions in some detail of the pro- 
gramme or project for which Swedish support is requested. The description should preferably 
contain clqar statements of main and subsidiary objectives as well as productioa targets for the 
various elements constituting the programme* or project. As a rule Swedish resources are contri- 
buted in the form of credits and granjs comprising funds, goods, consultants and staff. 

The volume and objectives of Swedish international development assistance are established 
by the Swedish Parliament and Government. Directives are given in the annual budgets and elabo- 
rated in various policy statements. On the basis of these directives, the Swedish Inteniational 
Development Authority, SIDA, prepares and implements the Swedish programme of development 
assistance. In its work SIDA co-operates with the Ministry of Foreign Affairs, to which it is 
administratively related. 

For Sweden's principal development partners, SIDA^s long term planning of bilateral 
Swedish development co-operation takes the form of special assistance programmes, so called 
country programmes, based on economic, social and political conditions in the individual countries 
as expressed in their development plans and other policy documents* (For other countries SIDA*s 
planning proceeds from existing motives in each single case). 

The country programme is worked out in close collaboration bet:^een the recipient 
Government, SIDA, the Swedish Ejnbassy and SIDA*s locai Development Assistance Office. Other 
bilateral and multilateral donors are consulted in order to co-ordinate the Swedish progranime to 
those of other development partners. Regular Swedish planning missions are a normal feature in 
this process. J^>f ^^ 
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The choice of sectors to be supported is guided by priorities established by the developing 
country. Reference is also made to Sweden's own priorities and potentials of providing the required 
type of assistance. Specific programmes and projects to be supported are selected within the frame- 
work of the country programme which is reviewed periodically. 

« 

ADMINISTRATIVE PROCEDURE. p" 
Requests for development assistance are dealt with and assessed by SIDA. Final proposals for 
Swedish support to development programmes and projects shaU be presented to the Director General, 
ihe Board of Directors of SIDA, and later, in applicable cases, to th^edish Government for 
decision. The progg^l takes the form of a memorandum prepared by the SIDA staff describing the 
progran>me or project to be supported. A positive decision requires that a number of criteria have 
been satisfied. 

The following is a summary of the content of a support memorandum. 
THE SECTOR 

A brief description of the sector to which the programme/ project belongs. The importance of the 
sector in the country's economy and its priority in the national development plan. A brief 
description of subsectors and relations to other sectors. Relevant historical, political, economic 
and social information.. Prevailing constraints and potentials for development. 

The sector's place in the national administrative system. Responsible organization and 
institutions. Relevant legislation. Administration at regional and local levels. 

Infrastructural assets, shortcomings and needs. Manpower resources and planning. 
Existing and planned education. Proportion of expatriate staff at medium and top levels. Research 
situation. « ^ 

Existence of pilot projects. Activities of other donors within the sector. Mechanism for 
local co-ordination of assistance. 

Previous Swedish experienc'e within the sector. 

TUB PROGRAMME OR PROJECT 



Previous Experiences 

Is the proposed project or programme a continuation of an on -going activity? ii so, describe as 
precisely as possible to what extent earlier plans have been implemented and stated objectives 
achieved. Analyse the deviations betwee^n plan and reality. 

116 

O 114 

uc 



Swedish criteria for programme and project support 



For projects and programmes without a past histor}* examine what experiences from similar 
activities are relevant to the planning. 

NEEDS AND OBJECTIVES 
Analysis of needs 

Which needs is the programme or project meant to satisfy? Are these needs discussed in the 
national development plan? In connection with manpower planning? Quote^authoritative comments. 

To what extent is the programme or project expected to satisfy the needs in question? Are 
further measures required now or later? Should this be investigated now? 

Definition of objectives 

Definition of objectives should clearly and concisely show what and how much is to be achieved for 
whom, where, when and how. 

Degree of achievement should preferably be measurable, i. e.^ be possible to express in 
. figures,, In cases where this is not possible, observations of effects should make it possible to 
describe what has happened in relation to stated objectives, for example that more or less of some- 
thing exists as a result of the change that has taken place. 

1. Main objective 

Which is the main objective? Which priority does it have in the national plan? Which are the target 
groups? Age, income, education, occupation etc. ? 

2. Subsidiary objectives 

List the changes which are intended to achieve the main objective, The changes should if possible be 
_ expressed as subsidiary objectives with intended quantities and/or qualities defined. 



^ DESIGN 

Describe the programme or project with reference to thfe following. 

- Main activities (teacher training, primary education, construction of 
^ secondary schools, curriculum development etc. ). 

- Activities and production process to be organized and implemented in 
order to achieve stated production targets. 

- Time schedule. Relevant illustrations including network of activities, 
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APPRAISAL 

Summarize the appraisal with regard to each of the following aspects: 

- Socio-economic aspects; 

- Sector techniques in relevant field of activity; 
, - Transfer of knowledge; 

- Construction and engineering; 

- Organization and administration; 

- Costs, timing, and financing; 

- Legal aspects. 

List the intended positive and potential, or unavoidable, negative effects with reference to stated 
objectives and the aspects appraised above, particularly the socio-economic aspects. 
Examples of guiding questions: 'v 

Does the programme or project have direct effects (positive and/pr negative) with regard to 
standard of living (food, clothing, housing etc. )? For whom? When? Indirect effects? 

Will the programme or project lead to increased govemment or private resources? Is there any 
indication of how they will be distributed? 

,Can the programme or project be expected to generate savings? Fpreign exchange? For whom? 
When? 

Which negative effects will be caused and/ or avoided if the programme or project is implemented? 

Do the goods produced have a market value? How has this been taken into account in the socio- 
economic appraisal? Have the internal economic and financial returns been computed? 

What are the expected effects on employment? On the eavironment? On population? 

Is the programme or project a precondition for other programmes or projects with the same or 
similar main objectives? (Road projects and power projects are examples of such preconditions). 

Will the programme or project contribute to knowledge, skills and attitudes favourable for ^ 
continued development? ' 

Do the expected positive effects outweigh the negative? Do the expected be^iefits jv»stify the costs? 



Appraisal is something done by a visiting SIDA mission appointed for the purpose. In sucli 
cases the mission works in consultation with the recipient^Government, the Swedish Embassy and 
SIDA»s Development Assistance Office. Before returning to Sweden, the mission in a meeting 
attended by interested parties reports verbally about its findings and tentative conclusions. On this 
occasion a written summary statement by the appraisal mission may be handed over to the Govern- 
ment departments concerned in order to facilitate further co-operation. Such a statement does not, 
however, in any way prejudge the view or decisions of SIDA or the Swedish Government. 



ALTERNATIVES 

Which alternatives to achieve stated objectives have been considered but rejected? Why? 
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EVALUATION , ' 

How is progress towards the stated objectives to be followed? 

Summarize the established planar evaluation both as regards periodic reports and special 
studies focusing on plan implementation. 

Examples of guiding questions: 

Which are the objectives to be evaluatea? Which institution or experts will make the evaluation? 
During what period of time? Which data mu/>t be collected before the evaluation can start? 

Should special studies be carried out regarding potential negative effects? Which?. How? When? 

Can the evaluation be co-ordinated with ^ther studies within the same field of activity? 

How is data collecting and reporting organized for built-in evaluation? Which production results 
will be included in the reporting? Which results will be followed up in some other way? How? 



THE SWEDISH SUPPORT 

In this section the recommended Swedish support is summarized. 

Reasons for the recommendation are offered with reference to the needs and priorities of 
the recipient country as well aa Swedish development assistance policy and available Swedish 
resources. ' ' 

Essential problems and rifks are, assessed with regard to objectives, time schedule, cost 
and financing. 



CONCLUDING REMARKS 

It should be pointed out in conclusion that the criteria and methods outlined above are meant to be 
adopted in a flexible and pragmatic manner. * 
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PROBLEI\IS OF. EVALUATION OF UNIVERSITY DEVELOPMENT 
CO-OPERATION PROJECTS 



by A. Kater andL.F.B. Dubbeldam 



4 

1. Introduction 

In this paper an attempt will be made to describe the problems of evaluation that are faced by Dutch 
Universities in their aid activities insofar as such activities are corned out under the Netherlands 
Programme for University Projects of Development Co-operation. 

The programme is still very new. It is only since 1970 that the Dutch national budget has 
included an item for a special programme for University projects. Previously, university actnities 
had been financed under the general technical-aid prograrpnT e of the Government or under other 
programmes. As the Minister for Development Aid allowed the programme and its projects to have 
a distinct 'university character' different front other government-financed programmf s, this meant 
that objectives, selection, procedures, evaluation, etc., were to be different from those applied in 
^ther programmes. New ways had to be found. 

The first major problem encountered arises from the fact that a number, of different bodies 
and individuals are concerned with the programme, namely: 

, (a) The Ministry of Development Aid - the financing agency - which suppo- 

sedly has developed its own routine of project evaluation in other 
programmes. 

(b) The Netherlands Universities Foundation for International Co-operation 
(NUFFIC) to which the administration, supervision and guidance of the 
Universities' programme has been entrusted. 

(c) The Universities, with their individual differences in academic potential 
and interests, attitudes toward developmental and international problems, 
organizational abilities, etc., each composed of bodies and individuals 
with their specific characteristics. 

(d) The Ministry of Education and Sciences. 

(e) The Ministry of Agriculture and Fisheries. 
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The parties concerned hold different views on the aims of the University programme. As diffe- 
rences of opinion on objectives have their consequences with regard to the priorities, selection- 
criter'a. evaluation, etc., we should mention the five most important views: 

(a) The programme should aim at using the Dutch academic know-how in 
the Dutch technical assistance - a governmental point of view. 

(b) The programme should enable the Dutch universities to fulfil their 
duties towards the Third World, as formulated by the 'Utrecht 
Conference 1967' of the Dutch universities. 

(c) The programme should enable the universities of the Netherlands to 
materialise their solidarity with the universities in the developing 
countries. " • • " 

(d) The programme should provide the Dutch universities with the oppor- 
tunity to carry out scientific research that has too much a pioneering 
nature to be included yet in the official policy towards research on 
the development problems. Views (b) and (d) express the ideas com- 
monly shared by the universities. 

(e) Finally it is believed that the responsibility for technical aid, including 
aid from universities, should rest with international organizations and 
that, as long as such multilateralisation has not been effectuated;- uni- 
versity aid should remain under thp direct responsibilitybf^the ' ' 
Minister of Technical Aid. This view is held by the Office-%f Inter- 
national Relations of one of the Dutch universities. ' 

Furthermore, practice has shown that many projects are the result of individuarcontacts between 
professors, in the Netherlands and the developing countries concerned, who saw an opportunity to 
carry out existing plans - if necessary with some adaptations - under the Universities' programme. 
If we may add here that also the participants of the projects have their own objectives and attitudes 
towards development problems and the projects they are engaged in, we believe that we have suffi- 
ciently indicated that there exists a complex and sometimes confusing multiplicity of objectives, 
which makes it very difficult to evaluate the programme in any phase. So far there seems to have 
been hardly any need for detailed evaluation techniques and orocedures, as the sum total of the 
projects applying for financing under the programme has been in balance with the amount of money 
allocated. However, at present the amount needed for potential projects has surpassed the money 
available (for 1972 about Fls. 6 milUon). In addition, the Netherlands find themselves in a period 
of retrenchment of Government expenditure. .For that reason the need is felt very strongly now to 
come to a more concrete agreement on the objectives of the programme and on a revaluation of 
procedures, ^selection-criteria, outlines for project- strategy, evaluation of the needs of the coun- 
tries concerned, and evaluation of project implementation and of results. 
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2. Evaluation of the educational policy 
Policy -makers are aware that an aid project makes sense only if a distinct and well-defined need 
exists in the country to receive aid. The^Government's technical aid programmes ana similarly 
the Universities' programme demand that proof be given that the proposed project will contribute 
to the socio-economic development of the receivingxountry. In general, the financing agency is 
satisfied with references to a plan for more than one year, a survey of an international commission 
and, or a letter in which the Government concerned declares that the project is welcome and fits in 
with the Covornment's own plans. 

Although we have no exact data to prove it, we believe that for a long time educational 
projects have been financed on the basis of the assumption that education in itself is capable of 
promoting socio-economic development. It was believed that educational policies that stimulated 
the growth of the educational system should be supported vvithout investigating the consequences 
for the other sectors of the social situation. Most educational aid was channelled through the acti- 
vities of voluntary agencies (usually missions, sometimes other institutes, like universities) or 
international organizations. Within certain limits, the fields of activities and the priorities pro- 
posed by them were accepted. As far as we know no sectoral or general evaluation of the educa- 
tional policy in the countr^ies that received aid has been undertaken by a Dutch aid-financing 
organi/jr.tion. , ^ , 

Often the actual evaluation of thejieeds of the country to receive aid is left in the hands of 
the Dutch partner in the planned co-operation, who is supposed to explain in the project- outline the 
needs to be met. Usually such an explanation is then presented as a letter by the institute to receive 
aid. This letter also serves as a 'request by the other party*. Sometimes an evaluation of the 
educational policy and situation ib available, either specifically prepared Ifor a certain project or 
for other purposes. Practice in other Government programmes includes that experts on education 
in a certain country, whenever available, are asked for advice on an application in the light of their 
experiences in and knowledge of the country and subject concerned. In general the Government 
demands that projects should contribute to the socio-economic development of a cou.itry. Th<? ques- 
tion, however, is whether, how and to what extent education contributes to socio-economic develop- 
ment. And then a satisfactory answer leading to applicable criteria for selection of priorities is^ 
very difficult to find. For one thing, who is to judge what is right for the developing country con- 
cerned? The donor or the country itself? 

Another niatter is that the assumption that education per se would promote development 
has graduall> become subject to critical review in, publications of the past few years. Consequently, 
mere transfer of knowledge to as many people as possible is no longer the best thing one can think 
of. Phenooiena like the increase of the number of unemployed school-leavers, for instance, have 
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made some policy-makers reconsider their policies. In this respect the universities have a distinct 
task. They could, in addition to the transfer of knowledge, whenever asked for. contrioute to more 
criUcal and creative thinking aniong the university students and staff of developing countries, For 
the problem does not seem to be that the university staff and students in the developing countries 
lack intellectual potentialities, Lat that the socio-economic circnm dances and the prevailing educa- 
tional systems have often failed to develop in them scientifically inquisitive thinking. Where such 
a situation exists, the Netherlands' Universities could- contribute not' only by teaching subject- 
matter, but also by stim .ing an awareness of the .problems, critical analysis, aptness in problem 
solving and ability to use the tools or resources available. 

id 

3. The machinery of evaluation of educational policy 
Except when it so h^appens that a sound evaluation of the educational policy and situation, in .a deve- 
loping country is available, neither the Government nor the Universities have the orgamzational 
machinery-to produce a thorough evaluation of any cour try in a reasonably short time. In the 
Netherlands there are quite a few institutes that are regionally specialized (like the Africa-Study 
Centre at Leiden, or the Documentation Centre for Latin-America at Amsterdam) or subject-wise 
specialized (Centre for the Study of Education in Changing Societies at The Hague). However, 
rarely do these institutes have adequate resources to keep themselves up-to-date about all the 
countries concerned, nor do they have sufficient staff and funds to survey the educational policy of 
a number of countries sufficiently thoroughly and quickly. 

Government representatives abroad generally are not equipped for this aspect of foreign 
relations; often their contacts, with the Universities in ccumry. and with the Ministries of 
EduoMtion and Community Development are infrequent. They are by tradition better equipped for 
external affairs (international politics, trade, financing, etc. ) than internal affairs like educational 
or developmental policy. Officials of the Dutcl. Foreign Service are. as a matter of principle, 
transferred frequently, so that they have little opportunity to familiarise themselves with the deve- 
lopmental problems of a particular country. So the Government foreign representatives cannot be 
expected to be a source of adequate information on educational policies. 

Voices nre heard, however, pleading for the appointment in the developing countries of 
special representatives to take care of the interests of universities (and other - educational - aid- 
giving agencies) in affairs of development co-operation. This, however, seems only feasible if 
there are sufficient projects in the countiy (or region) or if the task could be combined fruitfully 
with other work. The question remains what the costs of such a representative may be with regard 
to the total volume of the projects in the country (or area) to defend .the appointment. 
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This leads to the issue of concentration. A few years ago the Dutch Government decided 
to Umit most of the Dutch bilateral assistance to a certain number of countries. The Universities' 
programme is not Umited to the same countries. Yet, in order that the activities of .the univer- 
sities may be better prepared, implemented and evaluated some people plead for a concentration 
poUcy in the University programme as well: formatlv,,* of consortia of universities on the Dutch side, 
and/or concentration on specific countries or even concentration on a few universities in th.e deve- ' 
loping countries (trusting that the latter, in tlieir turn, may stimulate the other universities in their 
country). Some developing countries prefer s^ch programmes. Indonesia would Uke to see priority 
to be ^ven to ai^ to one of its five leading universities, while a number of countries have founded 
boards that have to judge whether the subjects of proposed research projects are among the prio- 
rities of the countiy. 

4. Priorities and selection of projects 
As mentioned above, the. ambiguities in the objectives of the programme make it difficult to roach a 
unanimous agreement on the criteria for deciding which projects may be admitted to the programme, 
and to what subject, countiy, etc. , priority should be given. . 

far the issue of priorities has not been urgent. To match the funds available the number 
of projects could easily reduced on the basis of such criteria as; is the project Ukely to contri- 
bute to the socio-economic development of the country? Does the subject fit in with the needs of the 
partner^ Does the project fulfil a set of general (administrative and preparatory) requirements set 
in advance to projects that may ht^ financed under the programme? What is partner's own contribu- 
tion to the enterprise? etc. 

However, the discussion on the criteria for selection and priorities has been reopened. 
The first step to be taken now is to try and reformulate the objectives of the programme in such a 
way that the number of potential projects will be automatically reduced from the outset; ,simult8- 
neously it is hoped that some of the criteria to be set may serve as guidelines for the preparation of 
new projects. Vital questions have to be discussed, such as: 

(a) Should concentration be pursued? If so, should the required »contribu- 
tion to socio-economic development' be maintained or is it possible to 
compose a 'packet' .of projects, including some academic subjects 
which are highly valued by the authorities in the developing countries 
but which do not visibly contribute to socio-economic development 
(e.g, law-reform, history, linguistics, etc.)? On the whole, which 
are to be preferred: projects that have sprouted from personal profes- 
sional (and professorial) contacts and are implemented - somewhat 
individually - with commonsense, with feeling for the problems and with 
de^iication, or projects that are part of concentrations or are fully inte- 
grated in and planned by universities (or faculties)? 
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(b) Would it be possible and desirable to give priority to projects that - 
e.g. as parts of a 'packet' - may lead to continued co-operation to the 
benefit of both partners after the aid-phase has been completed, rather 
than to co-operation projects that end at the moment of 'mission 
accomplished'. 

(c) What kind of 'piire^ research' should be financed by this programme, 
and what should be referred to other agencies? Or should research be 
accepted only if a training element (techniques, methodology, evalua- 
tion of facts, etc. ) is included? 

(d) Should projects that are part of international co-operation on the 
, 'donor' side be preferred to 'national' projects? 

^ (e)' How many criteria for priorities and selection and which of them should 

be formulated? Finally, the more requirements are set for projects, 
the more elaborate and refined the evaluation system should be. 

At present applications for financing under the programme can be submitted to NUFFIC once a year. 

After a first selection on administrative and procedural grounds, the proposals are discussed by 

panels, composed of people who are considered to be experts both on the subject matter and on 

development problems. Finally, the approved projects are recommended by NUFFIC to the Minister 

concerned, who, so far, has accepted the advice given and financed the projects. 

5. Control of preparation of projects 
At the moment neither NUFFIC nor the Government have adequate control procedures for the imple- 
mentation of projects. After the selection as described above has taken place and advice has been 
received frpm the officials concerned in the developing countries and after an incidental check -iip 
on the spot, the project can go ahead. Again, it is a ^oint of discussion whether concentration and 
Iqcal **f*presentation may make a more satisfactory control system possible. 

There is also the question of how much time and money should or may be allocated to 
evaluative (maintenance) research. Last year for instance the Government asked the Centre for 
the Study of Education in Changing Societies (CESO) for advice on an application fcr an additional 
subsidy for an extension to be built to a primary school in an African country. The way the appli- 
cant defended his case, the wa> the school was described (selection procedure at entrance, teaching 
programme, contact with the environment, staff recruitment, guidance of school-leavers, etc.) 
could not but lead to a most positive advice. Should not such a case be thoroughly investigated, 
either to prove the applicant has written 'empty words' or to observe, describe and publish a 
promising example? 

The same line of thought is pursued in the ideas on and plans for developing country wise 
central documentation, emphasising subjects like education, science and technology, planning and 
administration, all in view of development. The Royal Institute for Linguistics and Anthropology 
(Leiden) already has a resident representative for documentation purposes in Indonesia. 

124 



Evaluation of university development co-operation projects 



6. Evaluation of results 
The Universities' programme requjf es that those responsible for its projects report regularly on 
the progre^ss made. Summaries of these reports are published in a Progress Report of the pro- 
gramme as a whole and sent to the Minister. However, the question has been raised whether this 
way of reporting, supplemented by incidental visits of officials to projects, wui prove to be suffi- 
cient to serve the purposes of evaluation. For evaluation shouli5 lead to a set of results: firstly, 
it is a means td check whether the project has been implemented according to its objectives; 
secondly, the information obtained is supposed to serve as a feed^ back for the improvement of 
future projects and for the refinement of the objectives, the criteria and the organization of the pro- 
gramme as a whole. > 

For this purpose, NUFFIC has requested CESO to make the necessary studies of univer- 
sities' development co-operation and to contribute to the reformulation of objectives, criteria aftd • 
evaluation procedures. In 1971 a planned observation 9n the spot (with guidance) of a then newly 
starting project had tp be postponed because of staff and budgetary restriciiuxis. Yet we believe 
that a continuous observation of an actual case of co-operation is the best possible method of evalua- 
ting the impact of a project from a social, economic and especially pedagogical point of view. For 
no project is carried out in isolation. Staff of an institute that is to receive aid Uearn' from the 
project, but also from many other sources. They may internally select and re-integrate"^ments 
of previous experiences with those from the project concerned and from conferences, visits, talks, ' 
magazines, etc. Mere registration and comparison of the pre-project and post-project situation 
may not g^v^ sufficient data about the impact of one's own activities. A question is how many pro- 
jects cctn and must be 'observed' in this way in order to draw somewhat more gencraiiy valid conclu- 
sions as to university co-operation for development and in order to develop a set of new and efficient 
techniques of evaluation for future use. It would also be instructive to learn to what extent the 
broader programme objectives and the more narrow project aims match in reality; one can think oi 
situations where circumstances enforce a change in one of them, without necessarily leading to a 
premature end of the activities. 

For its plans CESO is considering finding local partners to co-operate with, in areas where 
many Dutch educational (university) projects are being executed. Such co-operation would be a base 
for CESO's observations of projects of university co-operation. 
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7, Conclusions 

Looking back one gets the impression that much of the Dutch educational aid of the sixties has been 
given in a somewhat opportunistic way (which does not imply that this has yielded poor results). 
The Universities' Programme of Development Co-operation is, very carefully, trying to develop an 
evaluation system that may cover all phases of the projects. As no now projects will be approved 
for 1973 as a result of budgetary restrictions, it has some time available to reconsider its policy, 
and it hopes that by the time the applications for the 1974 projects are submitted, it will have more 
clear objectives, criteria and evaluation routine's at its disposal. 
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THE ESFAHAN WORK-ORIENTED ADULT UTERACY PILOT PROJECT 
A COMMENTARY ON EVALUATION 



by J . Ryan 



1. Preface 

As indicated in the title, this paper is a cominentar;> on the evaluation of the Esfahan Wo;:k -Oriented 
Adult Literacy Pilot Project l/. It is based primarily upon the following three papers prepared fur 
the 'Seminar on the planning of out-of-school education' held at the IIEP during December 1971: 

M Bazany - ^Evaluating an experimental functional literacy project : the Esfahan 

\ experience^ 2/ . 

C. Bonnani - 'Description of the principles, methods and techniques adopted by the 
Esfahan functional literacy pilot scheme in planning and developing its 
curricula'. 

J, Smyth - 'Costs- effectiveness report on the work-oriented adult literacy pilot project 
in Iran : synopsis'. j 
Bazany is the Evaluation Speciaust assigned to the Esfalian Proiect. Bonucua had previously served 
the Project as Literacy Materials Specialist. Sm;yth's paper is based. upon a consulting mission to 
Iran undertaken for the Departm'ent of Educational Planning and Finance. No attempt is made to 
summarize the findings reported in these papers. Thp purpose of the commentary is that of 
reviewing the concept upon which the Project was based, certain of the analytical techniques 
employed and some of the more interesting evaiuational findings obtained. 



1/ What is referred to here as the Esfalian Project is, in fact, a sub-project 
within the Iranian programme- A related project is being carried oh in the 
De.zful area in south western Iran. 

2/ The results reported by Bazany and commented upon in this paper are 
typically based upon sample findings from selected programmes. Such 
results are thought to be typical of Project activities as a whole. There is, 
however, no statistical assurance oh this point. 
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2. The roles of evaluation 
A distinction is frequently made between the »formative» 1/ and »summative» 1/ roles of evaluation. 
The former involves the use of evaluative findings for programme improvements. The latter seeks 
ito make more ultimate judgements about the effectiveness of programmes in attaining their intended 
objectives. 'Summative' evaluation thus presumes the existence of accepted evaluative criteria. By 
contrast, 'formative' evaluation ne'ed make no explicit assumptions regarding standards to be 
achieved. While 'formative' evaluation does not preclude - and may often assist - 'summative' 
evaluation, the distinction in purpose is fundamental regarding the conclusions that may be reached 
from evaluational results. 

5" Functiona l literacy : The concept and its implications for 'summative' 
evaluation 

The evaluation of the Esfahan Work-Oriented Adult Literacy Pilot Project is predominantiy 
'formative' in purpose. This is, in a sense, ironic as the Project and the Experimental World 
Literacy Programme, of which it is a component, are defined and justified in terms of the develop- 
ment objectives pursued rather than the educational means employed. Both Project and World 
Programme are based upon the concept of 'functional literacy training' 2/. This newer form of 
training is differentiated from the more traditional kind by its emphasis upon relating literacy 
methods and content to economic and socixL development. More specifically, the designers of the 
Esfahan project perceived a shortage of literate and skilled workers to constitute a constraint upon 
economic de\>elor ment. Functioned literacy training was conceived as a means of loosening this 
constraint by upgrading manpower qualifications and thereby contributing to development. The test 
of this hypothesis Unking training to development would constitute an element of 'summative' 
evaluation. , 

It is evident that a number of prerequisites and conditions must be fulfilled before such a 
test becomes possible. Foremost among these is the development of an instructional technology. 



1/ This distinction is made, for example, by M. Scriven, "Metliodology of 
Evaluation" in AREA Mono^r^inh Series on Curriculum Evaluation, Vol. 1, 
Chicago, Rand McNally, 1967. 'Formative evaluation' takes place in the 
research development stage and is directed toward the improvement of the 
product under development. 'Summative evaluation' measures a product in 
relationship to pre-established objectives. 

2/ For a discussion of the concept of 'functional literacy' employed in the 
Experimental World Literacy Programme, see J. Cairns, Convergence, 
-Vol. Ill, No. 2, 1970. ^ 
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To date, this is the task that has preoccupied the Project staff. Othei requirements pertain to the 
research design. Theoretically, the exact linkage between the selected fields of training and the 
intended development consequences mu^t be specified. Subsumed under this task is the necessity of 
defining indicators of the aspects of development it is intended to affect and the obtainment of pre- 
l^easures against which post-measures may be compared. 

The measurement problem is further complicated by contemporaneous changes in factors of 
production other than the quality of labour. Regarding this last point, it should be noted that the 
project site, Esfahan, was particularly selected in order to observe the consequences of combining 
a manpower strategy with large-scale investments in agriculture and industry. The hoped for 
synergis^tic reaction, if realised, will ^^oth promote development and, incidentally, complicate the 
measurement process. 

A hypothetical example based on a Project programme may suggest the method and intent 
of 'summative' evaluation. The sugar beet training course provides a relatively straightforward 
example. Its purpose is presumably that of contributing to agricultural productivity by increasing 
the output of sugar beet. Actual production, if known, would be a suitable indicator. An observed 
increase in sugar beet production following the implementation of the training course is a necessary 
but insufficient procf of i>iogramme effectiveness. Evidence that v^ourse participants adapt new 
practices advocated in the training course and experience a more rapid increase in production than 
non-participants is more compelling proof. Production outcomes are thereby related to course 
participation through the plausible intervening variable of adaptation of improved practices. A still 
more convincing mode of proof would be constituted "by the finding that observed increases in 
production are better explained by course participation than by rival hypotheses: sale price, land- 
holdings etc. 

It may be maintained that such 'purist* standards of scientific rigour are not applicable to 
field operations While possibly valid, this contention is aside from the point. The issue is not 
between simple and sophisticated research designs, but x tither pertains to the role or purpose of 
evaluation. Is the intended outcome to,be a judgement of programme success in attaining its 
objectives, or is it, as with the Esfahan Project, the more preliminary goal of developing an 
instructional technology? In short, is evaluation 'summative' or 'formative' in role? 

4. The evolution 'of an instructional technology 
The problem of giving operational meaning to the cpncept of functional literacy became from the 
Outset the predominant concern of the Project staff. It was specified that training should be related 
to economic and social development goals and that it ought to be work-oriented. While of some help 
ill suggesting appropriate criteria, this definition specified neither the content of the training nor 
the nicthocfe of instruction to be employed. ^ 

130 

120 



Evaluation of educational systems 



A starting point in specifying programme content was that of determining the occupational 
skills in which instruction would be offered. Development plans (where available and applicable), 
knowledge of labour force structure and prevailing trends, and resort to commonsense judgements 
all played a role in determining such choices. Initially, the programmes were highly specific in 
naturp (e. g. the plant protection course). Subsequently, mpre general programmes were offered 
(e. g. the general agricultural course). The more micro-level decisions as to precisely what aspects 
of the selected topics should be emphasized and the order and manner in which materials could be 
most effectively presented posed a continuing, challenge. The present form of the Project's 
instructional technology - i. e. the selection and organization of content and the choice of methods - 
is the result of an evolutionary process. Earlier versions of programmes were replaced by 
improved ones. The role of evaluation was tliat of identifying programme difficulties and measuring 
the impact of proposed revisions. It was, in a word, 'formative' evaluation. 

The revision of the arithmetic units represents a highly successful example of the use of 
evaluational findings to improve course outcomes. In 1969. the order of presentation of material^as. 
the classic one employed in primary schools^ around the world: counting came first (one to ten in the 
first week, ten to one hundred in the second), next addition, then subtraction and finally multipli- 
cation and division. Evaluation showed this method produced vei^ disappointing results. It was 
observed, however, that in spite of low test scores, participants seemed capable of handling com- 
plicated calculations in their practical affairs (e. g. marketing). These findings and insights led to 
a complete revision of the 1970 curriculum. In essence, it involved the integration of arithmetic 
instruction with technical instruction. Arithmetic operations were introduced at the pomt at which . 
they were required for comprehension of the technical content. In many cases, the arithmetic being 
taught was a formalization and generalization of operations previously performed more or less 
intuitively. The improvement in test achievement was striking. In many programmes, the average 
test score was twice as high under the new method as under the old. 

5. Indicators of programme success 
The Project's two principal indicators of programme success are dourse participation and achieve- 
ment. The measures of participation are multiple: enrolment, attendance,, drop-out and punctuality. 
Examination score is the usual measure of achievement. 

Enrolment , 

The measurement of enrolment has both quantitative and qualitative' aspects. It is not only the total 
number of participants that is considered important, but also tlieir occupations. Separate quotas 
were established for women and for industry, agriculture and craftsmen. Actual enrolment shows 
little relationship to these prescribed quotas. Participation by women is nearly twice their allotted 
percentage. Farmers fill only 64 per cent of their quota and industrial workers and craftsmen only 
21 per cont and 12 per cent respectively of theirs. 
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The assignment of quotas relates to the functional literacy objjective of providing training 
fc>' cccupationb thought to play strategic rules in development. Situational factors probably explain 
part of the discrepancy between prescribed and actual enrolments. It is noted, for example, that 
industrial employers are generally not favourably disposed toward the participation of their 
employees in literacy programmes. While accounting fqr the relatively low number of industrial 
workers in tliese programmes, it raises more fundamental issues. Why should employers oppose a 
training course presumably capable of markedly increasing employee productivity? The willingness 
of women tc participate in large number^ poses an analogous issue. Apparently, the intuitive cost- 
benefit analysi-s of these women persuades them to invest their time in literacy study. In both of the 
above instances, there is an evident disparity between pilvate motivations and official enrolment 
priorities. Does this suggest that the latter are not soundly based? There is certainly neither 
utility nor nriOAcdity in offering education to the unmotivated while denying it to those who seek it. As 
the disparity between the assigned quotas and actual participation reveals, the Project has, in fact, 
adopted cui 'open door* policy, but seemingly finds difficulty in reconciling this sensible decision 
with official ideology. 

Drop-out 

Literacy courses are prone to relatively high djrop-out. There is neither the 'stick' of compulsory 
attendance nor the ^carrot' of a diplonrxa at the end of the course to assist in sustaining participation. 
Thus, the holding power of a course is a rough indicator of its utility as judged by participants. 
Drop-ojit »from the Project's courses shows an expected pattern. It is high during early weeks and 
then declines. It tends also to be high at 'decision points' such as between course stages^ The 
agricultural season and industrial work schedule have a predictable influence on both attendance and 
drop-out. The net result is that fewer than 50 per cent of the original par'ticipants finish a course. 

It would seem useful to make a distinction between drop-out produced by external factors 
and that resulting directly from the conduct of the course. In the short term, it is probably difficult 
to correct external causes of drop-out. The opportunity to work overtime or the need to care for a 
newly born Lafant typically have a pre-emptive claim on the time of participants. Drop-out produced 
by dissatisfaction with course content or scheduling may be more easily remedied. Unfortunately, 
the exteiudl factors are the more important ones, Bazany reports that 4^per cent of a sample of 
drop-outs reported work obligations as responsible fur drawing them from the course. Another 
19 per cent were driven from the course by dissatisfaction with its content. 

\^ 

5^* Achievement 

As discussed above, the (inal objectives of the Project are economic and social in character. 
Nonetheless, acHievement in literacy and numeracy constitute both means toward these more 
ultimate cuds and goals in themselves. Smyth reports that in the original request of the government 
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of Iran to the United Nations Special Fund, functional literacy was defin^ as equivalent to the 
achievement of the average 6th or 7t\ grade student. This was apparent^ later judged overly ambi- 
tious, as Bazany states that the present objective is achievement equal to t^at of an average 4th 
grade student. In 1969, sample results showed that the Project failed to achieve its objective. After 
the revision of course content and method referred to above, both the reading and numeracy object- 
ives were achieved in 1970. Writing remained below the prescribed standard. 

The achievement of such standards in a relatively brief training. period seems impressive. 
It is rather difficult, however, to know precisely what such findings imply. To begin with, many 
Project participants (probably 20 per cent in the programme in question) are classified as semi- 
literate at the start of their training. Even illiterate adults presumably understand many of the 
concepts that children must learn. They know, for example, what adding and subtracting imply even 
if they cannot accurately perform such operations. Moreover, the definition of achievement 
employed here measures but a fraction of the 'output' of schooling. It excludes certain of the 
^subject-matters taught, says nothing of attitude formation and neglects the custodial function that 
schools serve in all societies. This is stated not with the intention of minimizing the achievement 
of the Esfahan Project, but as a warning against the assumption that the 'outputs' of primary schools 
and literacy projects are in any sense directly comparable. 

Results on tests of technical achievement were less satisfactory than those for literacy and 
numeracy. Smyth attributes this to the characteristics of the teachers, few of whom are competent 
practitioners of the subject-matters in which they are offering instruction. Bazany notus that the" 
actual allocation of class time to teclinical content falls below the suggested limits. In tiie general 
courses in particular, the teaching of literacy skills appears the primary goal. Teclimcal training 
is a distant second in both attention received and results attained. 

6. Determina tes of programme success 

.- ' ^a 

For the reasons of both project management and research, it was necessary to identify those 

characteristics of participants, teachers and classes that bear a significant relationship to 

achievement. 

Characteristics of participants 
The findings on participants are interesting, even curious. Age bears a negative, but statistically 
insignificant relationship to achievement (i.e. younger participants, on average, are higher 
achievers than oldei participants). Measures of socio-economic status also appear unimportant. 
Li^teracy status upon entry into the course has a puzzling correlation with achievement. Uhuu the 
measure of literacy is based on a test, the relationship is negative, those who knew most in the 
beginning knew least, in the end. Conversely, when the initial measure derives from the instructor's 
j\tdgmnent, the relationship with final achievement score is positive. 
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The negative relationship between age and 'achievement is not surprising. While socio- 
economic status (SES) is usually strong and positively related to achievement in developed nations, 
it may be, or appear to be, less strongly so in developing so^^ieties. Indicators of SES are typically 
based on occupation and education. Neither measure is likely to yield a good statistical distribution 
in developing societies, particularly in rural areas. Public education has only recently become 
widely available. Most adult respondents will have had little if any schooling. Nor has occupational 
structure typically 'hardened' to the extent found in developed societu^s. The average rural citizen 
in a developing nation is a gereralist who shifts occupation with the turn of the agricultural season. 
The traditional indicators of SES are thus unlikely to be sufficiently well distributed to yield signifi- 
cant correlational results. Efforts are under way to find a more relevant and applicable set of 
indicators bf SES, i 

It is difficuj.t to know what to make of the conflicting evidence regarding the relationship 
between initial literacy status and final achievement. Do the initially better qualified students 
become bored at the slow pace of instruction and attend intermittently (or drop out) while their less 
qualified classmates participate with regularity and diligence? Such hare and tortoise outcomes are 
possible, but certainly unusual. As Bazany ohocrves, the teacher's rating of the participant probably 
includes motivation and intelligence and is thas not *:trictly a measure of literacy status. Being a 
more comprehensive evaluation, it proves a better prodictor of final achievement. 

Teacher characteristics anc class aiz% 
The evaluation results show a consistent relationship between teacher characteristics and the 
achievement of their students. Experience and age are negatively correlated with achievement, the 
teacher's educational level is positively related to student achievement. The explanation offezed is 
that younger, hence less experienced, teachers adjust more readily to the instructional methods 
employed by the Project,, 

Smyth reports that there is no significant relationship between class size and average class 
achievement. This might suggest that importa.*. economies could be achieved by increasing class 
size without detrimental effects upon achievement. The answer is probably no. First, there are 
physical constraints that must be taken into consideration. Rural programmes serve small and 
remote villages. It is thus difficult to either increase attendance from within the village or to 
combine the programme of one village with that of another. It should also be noted that increased 
class size could endanger future progress. Specifically, Bazany reports that while it was intended 
that group discussion be an important component of all instructional programmes, this objective has 
not been realized. Large class size could constitute an obstacle in correcting this shortcoming, 

A methodological note concludes this section. It will be observed that the relationship 
between achievement and its correlates are derived from bivariate analysis. That is to say the 
relationship between two variables is examined in isolation, in field situations where computers £tre 
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not available, this is perhaps necessary. It is also potentially misleading as social research deals 
with complex and interrelated phenomena. A relationship such as that between achievement and 
class size is Jjrobably conditioned by a number of intervening variables, in particular, mode of 
instructibn. Multivariate analysis improves upon bivariate analysis by permitting the simultaneous 
consideration of the influences of a number of variables upon a particular outcome. As such, it 
more nearly mirrors the complexity of reality and hence permits the more accurate estimation of 
relationships. 

7. Treatment of development costs ' 
The return on investments in educational developments depends both upon the novelty and worth of 
the 'product* developed and the ability of men to learn from experience. In principle, the treatment 
of development costs should reflect these same considerations. In fact, we.^o not possess and 
perhaps never shall obtain information on these issues. Accordingly, we must improvise a solution. 

Smyth proposes that ctie first three years of project acitivity. costing some $2,4 million, 
be treated as a development expenditure. Of this amount. Iranian authorities supplied approximately 
$1.3 million; the remainder wis provided by Unesco/UNDP, Amortizing the cost to Iran over a 
25-year period with an 8 per cfent interest charge yields an annual cost of $378 thousand. 

This is perhaps as sjfnsible a solution as any. One may quibble over whether the total cost 
should be included or only thJ Ir^anian portion and over the duration of the amortization period. 
Unless an extremely short tiLe period is selected (e. g. five or ten years), the last problem is 
unlikely to alter the annual cbst greatly. The principal difficulty is that an analysis of costs alone 
evades the issue of fundameiltal interest: what has been purchased? 

There is evidence dhat a viable instructional technology has been developed. Is this tech- 
nology applicable outside thl Esfahan area? Will it reduce the cost of 'producing' literate adults in 
the future? If time answer^ these queries in the affirmative, all of Iran's illiterate adults are 
potential beneficiaries of tl^fe project's development investment. There seems as yet essentially no 
evidence that 'functional literacy' represents a development strategy as well as an instructional 
technology. If it doe?, presumably all Iranians, not just present and potential course participants, 
are the beneficiaries of the investment made in Esfahan. 

8. Conclusion 



This paper makes a distinction between 'formative' and 'summative' evaluation. The evaluation of 
the Esfahan Project has been of the forme;' type. To date, the Project has been occupied with the 
development of an instructional technology. This represents an important achievement in its own 
right and an essential intermediate step toward £he attainment of the Project's ultimate economic 
and social goals. 'Summative' evaluation - i. e. evaluation of these more ultimate objectives - can 
and should now be begun. 
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PLANNING AND INTRODUCiNG THE EVALUATION- OF THE 
EDUCATIONAL TELEVISION SERVICE IN THE IVORY COAST 
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' ' 1. Objects of the programme ^ » 

The Ivory Coast Government has included in its Five-Year Plan 1971-1975, the renovation aiid rapid 
extension of primary education. Having in fact decided diat total enrolment for all children between 
6 and 11 years should be reached by 1986, it has dra\vn up a scheme characterised by the systematic 
use of television; in 1980, 16,500 clas,srooms, the total now existing, will he equipped with recovers 
and will take 720,000 pupils. At that date daily programmes of seven hours will be put out for the 
bepefit of the six years of the primary cycle. Other programmes will be addressed to young people 
and adults, making J ele vision an instrument of continuing educatioa (with the support of other media 
such as sound broadcasting and the press). 

Tl^e object of this scheme ]s to remedy the existing weaknesses in primary education, 
namely, the wide disparity in enrolment ratios between different regions (from 5 per cent to 95 per 
cent), insufficient qualification of teachers, substantial wastage between the first and sixth year of 
primary studios, and maladjustment of curricula <-ind teaching methods. 

This Educfulonal Television Scheme is also designed to give the whole population an instru- 
ment of communication, a common language, spoken and read by all, a means of continuing out- 
of-school education, particulp.rl^ Indispensable for primary school leavers who do not continue their 
studies. It is therefore set in the general context of a. cultural policy of national unification, aimed 
also at intensifying the participation of the whole population in national devefopnient, the main Imes 
of which, as defined by the 1971-1975 Plan, are; the extension of agricultural modernization and'tlie 
betterment of the countryside, the Ivorisation* of the economy, the orientation of the training 
mechanism so as to give everybody the means of adapting themselves to the needs of the economy 
and social progress, b> sharing in the national enterprise of transforming the conditions and modes 
of life, thought and action. 

. The use of television is explained by the fact that the Ivory Coast has a higli quality tele- 
vision infrastructure which, in 1973, will cover the whole of the national territory and which has so 
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far been very much underemployed. This appeared to be a means of ensuring both the sufficiently 
rapid pi-ogress of basic education and the accelerated organization of out-of-school education open 
both to young people and to adults. Furthermore^ on the basis of existing experience of school 
television in Niger, Salvador and Samoa, it seemed possible to take into consideration the educa- 
• tlonal advantages of television; the presentation of concrete and no longer purely verbal elements, 
the diffusion of good audio-visual models, openness to the world, and the constant updating of the 
terms, content arid methods of teaching; the unification of teaching, the generalization of the work of 
specialized and highly qualified teachers for the benefit of the whole teaching body. 

This Scheme was therefore thought likely to lead to an appreciable improvement in the 
internal efficiency of prirriary-education (particularly by the reduction of repeating and drop-out) " 
and a better adjustnient of teaching to the needs of development, 

2. Organization 

The introduction of the Educational Television Scheme was preceded from 1967 to 1969 by a number 
of economic, sociological, educational and technical studies. It was started in 1969 by the install 
lation of a 'television complex* at Bouak^, tbe second city in the Ivoi^ Coast, in tlie centre of the 
country,- 400 km. from Abidjan. 

This complex, which has no equivalent in other countries and which is still housed in 
temporary accommodation, consists of: 

" a department for the training and refresher training of teachers consisting of a Teacher 
Training College which trains full-time teachers in three years v^ Xh a view to becoming 
headmasters, teaching advisers or to holding executive or specialist posts in tlie 
Educational Television Scheme, and a pilot CAFOP (there are seven CAFOP in the whole 
of the Ivory Coast for the training of ordinary primary school teachers, who are trained 
^ in one year). This training H^partment is also responsible for all r^efresher training of 

serving staff (teachers, headmasters, educational advisers, primary school inspectors, 
etc. )j 

' a publications centre, producing printed material in support of television programmes. 
- common services, especially a research and internal evaluation centre, associated with 
an operations service which ensure? the rapid feedbact^from the television receiving 
schools to the two production centres; a general and audio-visual library (documentation 
for all media) and a technical maintenance service. 
The personnel of this complex (educationists, producers, studio staff^and administrative staff) 
consists of experts supplied by external aid from a number of sources (Unesco, France, Canada) 
and of Ivory Coast nationals. 
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* The complex, under the authority of a Director who is also an Ivory Coast national, oomes 
under the State Secretariat for primary education and television education, associated with the 
Ministry of Education and responsible for all operations. The State Secretariat is also resppnsible 
for a service for the installation and maintenance of receivers in the schools and a publications 
distribution service. 

The Secretary of State for Educational Television has the benefit, in carrying out the whole 
of the Scheme, of the technical advice of a Scientific Consultative Commitee consisting of senior 
specialists who can speak with authority in the fields of education, communications and educational 
technology, selected by the Government from a number of countries (Belgium, Canada, United 
States of America, France, United Kingdom, and the Federal Republic of Germany). / 

Finally, the representatives of governments and multilateral institutions co-operate with 
the Ivory Coast in this large-scale undertEddng, meeting in a group for the co-ordination of external 
aid, known as the Abidjan Club. 

An essential charactexistic of this project is that it is supported by extremely varied 
sources; the World Bank, the United Nations Development Programme, Unesco, Unicef, the 
European Development Fund, France, Canada, United States of America an,d the Federal Republic 
of Germany, shortly to be joined by Belgium and Italy, in fact take part in this exemplary under- 
taking of international concerted action and associate themselves with the considerable effort being 
made by the Ivory Coast Government. From 1969 to 1973 the investments made in this operation 
amounted to more than 2 billion Francs CFA, external aid in operations (specialized personnel), 
amounted to 366 million Francs CFA for 1971. 

3. Present position and prospective development 
The Bouak6 corhplex was opened in October 1969, the Teacher Training College received a first 
intake of 70 pupil-teachers and the pilot CAFOP recruitcu 120 assistant pupil-teachers. During the 
same year, 1969-1970, the first production studios began to operate. In October 1971, in accor- 
dance with the planned tinietable, 446 CPl classes (first year of the primary education cycle) 
equipped with t^evision receivers, began to receive th(rprogrammes^(four/10 to 20-minute 
programmes a day) and to apply new curricula. 

^ Each year, 500 new classrooms will be equipped and a new level of ths primary education 

cycle will be affected by the reform and the introduction of television, going up to CM2 (sixth^year pf 
. primary'studies) in 1976-1977 by which date the number of television classes will have risen to 
10, 000, with a total of nearly 450, 000 pupils. 

It is planned that in 1979, the 6-7 age-group, which will then include about 170,000 children, 
will be admitted in full to the first year of remodelled television education, .and that within six years 
thereafter general enrolment will be achieved. 
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^ In parallel, the'^organization of out-of-school education acti\'ities should begin in 1^72 and 
■should keep pace with the developm4nt of televised primary education. 

4. Evaluation of the Scheme 

The need for evaluation 
In the^phase of preparatory studies the principle of evaluation of the Scheme was laid down. 

In the first place, the Ivory Cdast Government wanted to be in a position to assess the 
effectiveness of the reforms introduced into its primary education. The 1971-1975 Plan should, in 
effect, aU'ow primary education to develop, wifhout this effort prejudicing those who should benefit 
in priority from secondary and higher education and training. This means that the aims laid down 
for the Educational Television Scheme must be attained within the limits of the financial resources 
provided and that the basic assumption of this programme (substantial improvement in the internal 
efficiency of the system thanks to the use of television and better teacher qualifications) must prove 
to be well-founded. 

In addition, part of this evaluation must be devoted to the permanent review of the system 
through the organization of regular feedback. 

Furthermore, the various bilateral and multilateral external aid agencies engaged alongside 
the Ivory Coast Government in carrying out the Scheme expressed the desire to measure the effects 
of their co-operation and to see the justification bf their hope of finding, in this massive resource to 
the technology of education, the solution to the problems raised by the generalization of basic 
e^u^tion in the developing countries at an acceptable cost. 

Ever since its first meetings, therefore, the Consultative Group for the Co-ordination of 
.he Scheme (the Abidjan Club) decided that continuous evaluation would be carried out and that, in 
ord^r to guarantee its objectivity, it would be financed by support from governments and institutions 
not directly taking part in the Scheme. 

The objectives of evaluation 
This evaluation should, in this way, establish whether the Scheme has attained its objectives namely: 

- To give a better distribution to the education effort, so as to make the 
primary school a factor in national unification, both cultural and economic, 

- to link this effort to development projects, so that primary school leavers 
can integrate themselves in a dynamic environment, for good reasons and 
wilh advantage to themselves, instead of leaving for the towns, 

- to improve the educational, and therefore the financial, return on educa- 
tion, so that as many pupils as possible wh6 are admitted to CPl complete 
the cycle within a normal time after acquiring the basic knowledge deemed 
indispensable to the social and human betterment of all Ivory Coast 
citizens. 
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- to unify the remodelling of primary education by allowing the simultaneous 
generalization of reforms of content and method, which could not be done , 
even by a very substantial effort of refresher training conducted with 
traditional means. Thanks to television, the results of educational 
research will be applied very fast by the whole teaching body, a continuous 
teacher-traiping college.being thus made a reality. 

Measuring the results obtainr^id in ihe^e different directions amounts to verifying the proper working 

of the remodelled primary education system in liaison with out-of-school education, a system open 

to the national environmp-it (economic, social, cultural) and whose ultimate goal is the development 

of that environment. 

From this point of view, it must therefore be an overall evaluation made in terms of 
systems analysis at Ivory Coast level, with a view to assessing the impact of this system on the 
country. 

But an evaluation of this kind is also an instrument for the execution of the Scheme by 
enabling continuous adjustments to be made in the various sectors of: 

- management and administration, 
f - educational content and metliods, 

* • - technology, 

economics and financing, and cost-benefit ratio. 
It should also rf^late to the system as such, that is to say, as a combination of various means, 
methods and techniques and the choice of a ^strategy for the introduction of innovations deemed 
adequate to the objectives. 

Organization of the evaluation . 
In the light of the foregoing, a great deal of trouble was taken to fin'alise the organization of the 
evaluation, and it is, in fact, not yet finished, though it is possible to set out the main lin^Jigre. 

- A team responsible for overall (or synthetic or systematic) evaluation . This is also 
regarded as the outside evaluation assigned to specialists provided by the institution.^ and govern- 
ments which are talcing no dyect part in executing the Scheme (United States of America, the 
Federal Republic of Germany, and, probably in the near future, Belgium). 

^ This team at present consists of a permanent team-leader and consultants from the Institute 

for Communications Research (Stanford University), the^Staats Institut fUr Bildungsforschung and 
Planning (Munich) and the experimental education laboratory of the University of Li6ge. It is at 
present completing th<j working plan for the whole of the evaluation. It operates at the level of the 
State Secretariat for primary education and educational television. It is assisted by the management 
and administrative office of the Educational Television Scheme, also attached to the State Secretariat, 
which compiles a number of administrative and financial data.' 

- A research and evaluation unit , located at the level of the Bouak^ television complex, 
which is responsible for continuous educational evaluation, geared to the operational service, it is 
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responsible for ensuring permanent feedback between the new television classes and the centres for 
the production of television programmes and printed matter for teachers and pupils. The research 
and evaluation unit and this operating service are also in contact with the training and refresher 
training department of the complex. 

The elements of evaluation 
'Data on the management of the system. These data relate to the structure and functioning of the 
system: 

(i) Organization chart, information and administration circuits, legal 
structure, links between Ministries (co-operation among services 
concerned with the Scheme), co-ordination of external aid. 

(ii) persoflnel administration, status; methods of recruitment, training and 
advanced training; 

(iii) programming and administration of all operations (relations between the 
present system and the new system) and of the various sub-systems 
responsible fc^^^^aining and refresher training, television and radio 

' production, publhs ^tions, i nstallation and maintenance of the transmission 
network and the rec giving >tetyork, distribution of publications, research 
and operating service, buildings and equipment; management of stocks, 
equipment and premises. 

(iv) information of the public about the programme and its evolution. 
Technological data 

They relate to the production, transmission and reception of television and radio programmes and to 
the production and distribution of printed material and thus affect the television and radio pro- 
ductions* centres, the publications centre, the equipment of thelvor^ Coast Broadcasting Corpo- 
ration, receiving equipment in the schools and all the equipment used for P-ersonnel training and 
refresher training, both in the Bouak6 complex and in all the CAFOP. 
^ (a) Equipment used, suitability for its purpose; 

(b) effectiveness and reliability of use, full employment of means; 

(c) premises of the Bouak6 complex (temporary buildings, sOlans for perma- 
nent iDuildings); school prenaises (architecture and eq'utphccjent of television 
schools). ' ^ 

V E-ducational data 
The evaluation data in this field include those relating to: 

(a) The degree of cognitive and psycho -motcir development (the latter mamly 
centred on manual and graphic motility, fthe evaluation of which is useful 
for childi'-fcii beginning to learn towrite); 

(b) written and oral knowledge of French (the only language of instruction); 

(c) assimilation of mathematics; 
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(d) verbal interactions between teacher and pupil in4he classroom; 

(e) behaviour and attitudes of pupils in class; / 

(f) opinions and attitudes of teachers and parents; 

(g) teachers* knowledge of the' subjects taught (especially French and 
mathematics) on completion of initial training and refresher training. 

The populations affected by the elaboration of instruments of evaluation are the pupils and teachers 

who are completing one of the years of the primary cycle for the first time. In other words, this 

evaluation is at present being undertaken for pupils and teachers who complete CPl, the first year 

of primary schooling in 1972. 

The instruments worked out will then be applied to the teachers and pupils of the following 
year. .This operation |all lead to a better adjustment of the instruments to the real state of affairs. 

For the purpose of comparison, this action will be carried out with masters and pupils of 
television and non-television classes simultaneously. 

Stages in the construction of instruments 
An instrument of objective evaluation must meet the following requirements: 

(a) The subject matter of the evaluation must be clearly defined ^phases o"t 
the construction of tests, observation grids, questionnaires; 

(b) the instruments must be handled and the replies received must be 
corrected as uniformly as possible ( standardization phase) : 

(c) the results must be interpreted on the basis of norms established by the 
prior^ examination of a fairly high number of 8ubj[ects which allows each 
of the answers to be sited within a statistical distribution (phase of 
establishing norms) : 

(d) the data compiled must provide proof that the instrument really measures 
4 ;vhat it is supposed to measure ( validity study ); 

(e) ih^ test must produce the same results each time it is repeated ( fidelity 
stihjv)* 

To meet these requirements, Hhelfollowing procedure has been laid down: 

- Phase U ) A preliminary sample establishes the cultural range and linguistic 
capaciti'es of the populations to which the instrument is intended to apply. 
On the basis of this information, a first ^version of the instrument will be 
worked out in the Laboratory of the University of Li^ge.^, 

- Phase 2: A first application in the field to restricted samples will provide 
information as to the manageability of the instruments and indications of 
the difficulties and discriminations of the items of the tests. Modifications 
may thus be made and new items drafted. The instrument will be com- 
pleted by precise instructions. ^ 

- Phase 3 : It will then be applied to a representative sample. The analysis 
of itenis, establiighment of. norms and fidetlity calculations will be carried 
out at the University of Libge. This last pl^se will he repeated in the 
event of unsatisfactory results. The validity of the instruments 
constructed will also be studied. 

- Phase 4 : The instruments are now operational and can be generally applied. 
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Training of staff responsible for applying the iastruments and using the 
information 

Concurrently with the finaHsation of the instruments, staff will be associated with the various stages 
of the undertaking. In this way, on the expiry of the three years set aside for the construction of 
the instruments, they will be able to use them (tests, compilation^f data, coding, processing, 
analysis of results). 

It is nevertheless necessary to provide for the specialization and training abroad of certain 
Ivory Coast nationals, who will be called upon to take over from the experts. The Experimental > 
Educational Laboratory might assume responsibility for this speciaHzation. 

Financial and economic data 

(a) They relati to the state of expenditure and audit relating to all operations of training and 
refresher training, television and radio production, publications, transmission and 
reception, feedback, research and evaluation, as well as to the functioning of the" * ' 
primary schools and school inspectorates and of the central services. They cover both 
capital and recurrent expenditure, including the various external aids, but identifying 
them clearly. 

(b) They should make it possible to check or confirm the cost estimated made when the 
Scheme was formulated, and the regular updating of this Scheme. 

(c) They should also point to possible savings and provide the Scheme Management with 
elements of decision on this subject as well as estimates of the national budget and 
external aid. 

(d) Finally, these data should make it possible to establish unit costs per teacher and per 
pupil and their various components /cf. point (aj/ in Uie light also of changes observed 
in drop-out and repeating. 

(e) In addition, economic and socio-economic data relating to the impact of the system on 
the national environment should be coinpiled (the relation between the penetration of 
educational television and local development, ways in which this teaching is integrated 
in the active life of pupils, etc. ). 

(f) The foregoing data should be used for an approach to cost-benefit analysis. 

5. Overall evaluation of tho system 
On'the basis of the various sectoral evaluations set 9ut above, and of their synthesis, the overall 
evaluation team is responsible for determining: 
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(a) For the national authorities, the elements of decision which v/ill enable them to 

establish, pursue or modify the strategy for the conversion to the educational television 
system (both in-school and out-of-school), to adapt the management of this system to 
the need to obtain an optimum cost, as well as to the qualitative and quantitative results 
hoped from the reform of primary education and the organization of out-of-school 
education. 

In systems analysis terms, overall evaluation should integrate the various data obtained 
at the foUowing^different levels, in the light of their relevance to 6ach of them, and in 
such a way that all the operations carried out are self-consistent and consistent with 
the general education policy defined by the government and that continuous feedback is 
ensured between the implementation of the policy and the higher levels of programming 
and policy. 



General policy 
objectives recognized 
restraints imposed 



Needs and resources 
of subsystems and 
of the system 



Flow of material 
Flow of information 
Flow of decisipns 



determination of 
education policy 



programming 



processing 



implementation of 
policy 



(b) For external aid agencies, which are interested in th^first aspects of evaluation, the 
elements of decision which will enable them to extend the Ivory Cbast* experience to 
other countrij^Qs who want to apply these solutions, eitlier retainirtg the same strategy, 
the same plan of action and the same, type of personnel and equip tnent,. or modifying them 
so as to remedy any defects or shortcomings observed. 



6. Evaluation of the evaluation 
The overall evaluation group should apply a system of self-evaluation which Lould enable it 
constantly to adjust its methods, as well as to study the use made of the results of its investigations. 
It will be helped in this, as it has been in drawing up iis working plan, by the Internatiorial Scientific 
Consultative Committee for the Educational Television Scheme and the consultative group on the 
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STATISTICAL RESEARCH ON INTERNAL EFFICIENC^Y OF 
SCHOOL SYSTEMS 



by G, Carceles 



> 

Introduction , 



There is no longer any doubt as regards the need for systematic evaluation of school system functions. 
This obviously implies the quantification question, even in those cases where the points to be eva- 
luated appear to be mainly qualita^e. . 

The evaluation exercise aims at providing some kind of objective evidence of accomplishment 
in terms of outputs and to set this against the cost of the inputs. This process implies a quantifica- 
tion of goals and targets against which progress may. be measured and effectiveness evaluated 1/ 

The present paper will review very briefly the essentials of work undertaken by the Office 
of Statistics of Unesco • in close co-operation with interested Departni^ts and in consultation with 
the HEP - to evaluate the patterns of internal efficiency of educational systems so as to identify their 
major bottlenecks with an action-oriented approach, ^by suggesting simulation, exercises enabling the . 
policy-makers -to estimate the »price' and results of given measures. 

Three m^in phases can be identified in this task: (1) The preparation of the statistical 

ft ^ 

aspects of the 32nd session of the International Conference on Education (Unesco/IBE, Gciicva, 

July 1970); (2) Plarining the implementation of the statistical aspects of the Recommendations adopted 

by the ICE* (3) The follow-up. 

. * 

A. The 32nd session of the I. C. E. 
The International Conference on Education at the end of its 32nd Session (July 1970) adopted.its 
Recomnriendation No, 66, entitled "Improved effectiveness of education systems, particularly through 
reduction of wastage at all levels of instruction" in accordance with the terms of reference of the 
Conference, ^ 



iJ United Nations Institute for Training and Research "U.N. development 
aid, criteria and methods of evaluation'^ Nev^ York, 1971* 
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The Technical Commission discussed this topic and had before it a -working document 
"The Reduc tion of Wastage in Education '' (document ED/BIE/CONFINTED 32/4) and a .reference 
P^P^T^ " The Statistical Measurement of Educational Wastage '' (document ED/BIE/C0NFINT£D 3,2/ 
Ref 1). The reference paper was the result of a survey undertaken by the Office of Statistics in 
1969, with a view to evaluating the combined and separate effects of repetition and drop-out, as 
factors of educational wastage, and their incidence in the internal efficiency in educational systems. 
It should be emphasised that this was the first attempt towards the study of this very serious pro- 
blem at the international level. It is worth n^/ang also that the statistical paper took into considera- 
tion the points raised during a meeting of experts on educational wastage, convened by Unesco and 
the International Bureau of Education in Geneva from 10 to 14 November 1969 and at which a working 
paper stated the conceptual framework, assumptions and limitations for the evaluation of educational 
wastage. 

The 1969 Survey included all Unesco Member States and their territories. 148 replies were 
received but only 5 8 of them provided data on a complete enough basis to allow their analysis. 

The study's main aim was the applied presentation, in national terms, of a series of me- 
thods and techniques leading to the statistical measurement and localization of drop-out and repeti- 
tion, thus demonstrating the paramount importance for the planning of an efficient organization of 
national educational statistics. The main feature of this work is the derivation ol^ set of indicators 
of internal efficiency - related to each country's educational system - and treating separately each 
level of education as well as the two cycles of the second level. 

Two major results appear in this study: (a) There are two quite distinct features of wastage - 
namely repetition and drop-out - and they sometimes work in very different directions. Separate 
profiles for both repetition and drop-out are thus essential for meaningful system analysis as well 
as for the evaluation of the cost of wastage, here^studied in non-monetary terms, (b) The grouping 
of countries according ttTwastage patterns is often questionable, as the indicators derived have a 
concrete meaning when considered within the context of their own educational structure and should 
be very helpful if used to compare different regions or zones within a country, but would require 
attention if tliey had to be used for inter-country comparisons. 

Among the main items of the ICE . Recommendation appear the statistical studies to be 
carried out in this field with reference to the methods used in the 1969 Unesco Survey. More spe- 
cifically, it is stated: 

"Further studies should be undertaken, using the case-study methods at national level and 
and with international assistance. If possible, Unesco should take the lead in promoting these 
studies, in which regional educational offices and centres could take an active part. The studies 
should deal with the following points, the work done by the various regional organizations being 
taken into account: ^ m ^ 
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(a) How to achieve greater accuracy in the collection of data; 

(b) How to check the reliability of wastage indices and Qf the deductions to 
be made from them; 

\ (c)' 'rtie elaboration of techniques for the assessment of wastage in school 

systems without repetitions or drop-outs; 

(d) Elaboration of indicators of wastage for the purpose of simulation on 
the basis of alternative hypotheses; 

(e) The nature and incidence of wastage in higher education. " 
It is evident that this step forward aiming at the analysis of the educational system with a flow- 
oriented appVoach rather than the purely stock situations, raised a set of reservations. Some of 
those reservations are conceptual; others question the validity of the probabilistic model. and some 
of its assumptiortB Nobody questions, however, the value of identifying as explicitly as possible 
the characteristics of educational wastage, i. e. the patterns of repetition and drop-out in an edu- 
cational system and tlieir influence on the actual costs. * 

In fact, the search for a more efficient utilisation of means implies greater intake pos- 
sibilities without extra outlays This simplified statement is based upon the current fact of high 
holding power (theoretically positive) meaning disguised failure (i. e. repetition). V 

Whether repetition is^n ill or not, whether drop-out is or is not an indicJiJor of mal- 
functioning of the system at a gi\en stage are value judgements out of the purview Astatisticians. 
It is up to the educators and planners to qualify under specific conditions the meaning of these 
phen^ena, in some cases at the unrt level rather than on an aggregate basis. 

A In other words, it should be repeated that any statistical analysis of educational efficiency 
must b^ geared to a set of tests allowing 4 better insight into the problem of a comparative evalua- 
tion ofShe oufput from the educational system, at its various stages, considering both its optimum 
capacity and any existing plans or programmes on thii^* subject. 

Thus for instance, drop-out in a givenigrade may be a very logical result in cases where 
the relevant schools do not go beyond that grade. On the other hand, drop-out or repetition could 
indicate the existence of one or more critical points determining a deficient working of the educa- 
tional system Whether the factors are internal or external will be found only when its magnitude 
and location are established. Again, we have to repeat that the conclusion depends upon the yard- 
stick used, i.e. existing objectives and limitations. 

It Beems pertinent to state that studies on efficiency should cover not only the formal 
^ network of the^ educational system but also the increasingly important out-of-school facilities, to 
correctly assess the meaning of educational attainment of any population, in a development context. 
The above point is based upon a broader coverage of the data collection system if and when 
the countries themselves are able to provide such data. In this connection, the accuracy and relia- 
bility of the data (Point (a) of the statistical aspects of the ICE recommendation quoted above) are 
essential for any correct analysis of the problem and are the major concern in the Statistical 
Office's plan of work. 7 
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There is, moreover, an increasing awareness of the need for evaluation of school efficiency 
at the unit level with a view to clarifying the bases for internal imbalance which may n^^t be evident 
when averages are computed. Here, as in the case of the studies launched to assess differential 
patterns by regions or zones within a country, the main problem is the receipt of the consistent and 
reliable statistical data. 

B. The implementation of the ICE recommendati on 
Three of the four remaining ppints in the statistical aspects of the Recommendation appear to permit 
some immediate work, namely; 

(b) Checking the reliability of wastage indices and of the deductions to be 
made from them; 

^ (c) Elaborating techniques for the assessment of wastage in school systems 
without repetition or drop-6uts; 

(d) Elaborating indicators of wastage for the purpose of simulation on the 
basis of alternative hypothesis. 

As for the last point, (e) (the nature and incidence of wastage in higher education), it might be 

thought that only sampling studie^s in the field would be adequate. 

In order to consider the next steps to be taken for the implementation of the statistical 
aspects of Reconimendation 66 adopted by the ICE (32nd session, July 1970) an expert meeting on 
the Evaluation of Internal Efficiency of Educational Systems, was organized by the Division of Statis- 
tics on Education, at Unesco Headquarters from 17 to 21 January 1972 2/. 

Besides the continuous staff work on the above points, three major tasks were undertaken 
by consultants with a view to the development and verification of some of the methods and assump- 
tions presented at the I.C E, in the reference paper mentioned. This is particularly appUcable in 
the case of Mr. Berstecher's 2/ and Professor Hellwig's 3/ papers on simulation and prediction of 
wastage, respectively. Professor Brimer^s 4/ paper deals with the improved identification of 
wastage events. ^ 



ly Unesco, Summary of conclusions, expert meeting on the evaluation of 
internal efficiency of educational systems, Paris, March 1972. 

2/ Berstecher, D. , "Costing educational wastage", December 1971 
(Expert meeting on the evaluation of internal efficiency of educational 
systems, Paris, 17-21 January 1972). 

3/ Hellwig, Z. , "Wastage prediction method", December 1971 (Expert 
meeting on the evaluation of internal efficiency of educational systems,. 
Paris, 17-21 January 1972). 

4/ Briiiier, A. , "The quantificatioa,of school events related to educa- 
tional wastage", December 1971. 
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The simulation exercises referred to in the reference document put before the ICE by the 

UnescaOffice of Statistics are the main topic of Mr. Berstecher's paper ("C osting Educational 

f? 

Wastage - A Pilot Simulation Study '*) which after reviewing the basic concepts on costs uf wastage 

and criticising the conventional approach, develops a simple simulation model and presents a set 

of strategies Ccost-neutral*, ^medium-costly* and 'cost-intensive') on alternative bases. 

Two major points are investigated in Professor Hellwig's paper (" Wastage Prediction 

Method" ): 

(a) How accurately is the reaLflow of pupils re-constructed by the ' 
Standard Modefof Educational Wastage? 

Clearly, if the prpbabilistic flow constructed by using the observed promotion, repetition and drop- 
out rates were close to reality, the margin of error observed in simulating the actual facts (let us 
say, past enrolment at the various grades) should be negligible. The three methods presented by 
Professor Hellwig lead into such an accuracy of prediction that the> confirm the basic likelihood of 
work undertaken in this field. The application to the case of Colombia (the availability of dat^ was 
the criterion for choice of this country) showed very low margins of error, 

(b) To what extent does failure or success in a given grade in a given 
year depend upon previous performance? 

On this very important point - actually until now at the heart of the weakest assumption in evaluating 
wastage: the homogeneous behavioural pattern of all pupils enrolled in any grade which assumes that 
all pupils in^a grade, whether repeaters or not, have an equal potential for repeating that grade 
Professor He^Iwig recommends an independence test to determine differential patterns of promo- 
tion and repetition according to previous performance. In other words, the test aims at measur- 
itig the successive behaviour of repeaters and non-repeaters, by grade. 

Data allowing the application of the test were made avaiiaLle in the last few weeks for Chad 
The data covered the total of pupils enrolled in the last grade of first level. Tor three consecutive 
year (1967-68, 1968-69 and 1969-70) and distributed by age and sex and with detail of date of 
entrance tp each grade (i, e. their school career). (It is to be noted that since that population 
covered onl> pupils reaching the last grade, the conclusions to be drawn might not necessarily be 
valid for the overall enrolment, including pupils dropping-out). 

The preliminary work undertaken by the Secretariat leads to the following conclusions for 

Chad during the period under consideration: 

(i) Promotion and ^jepetition are independent events in which the latter 
does not affect me former, ^ \ 

(ii) This feature is valid for all gvades, whatever the age or the sex of 
the pupils. 

Moreover, a functional relationship was found between the age of entrance of pupils and their sue* 
cessive repetition rates, indicating higher ra^es in the case of pupils joining school at the earl>^ 
ages and decreasing consistently as age increased* 
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In a difjerent w^, another step forward in the determination of the validity of the assump- 
O" hoj:nogeneous behaviour was possible with data from Colombia. It is necessary to stress 
that the charice of having the required information from this country is most fortunate since it will 
^ allow the comparison of behaviour by age/ grade against flow analysis as included in Professor 
H^Uwig.^s paper * Thus the data available (Enrolment by age and sex for 1966 and 1967 mth indica- 
tion of newcomers) for first grade, allowed the derivation of an overall repetition rate and the 
breakdown according to the ages. It should be recalled again that these data are of a different type 
from the school career data of pupils in Chad. 

In the case of Colombia, for the years studied, the main conclusions are: ' 

(i) Repetition differs according to ages. Pupils entering at 7, 8, 12 and 
years register rates below the average, while at 9, 10 and 11 the 
rates are much higher. The peak is reached by pupils entering at 
11 years. 

(ii) There is no" difference in the pattern by sex. 

(iii) More than 50 per cent of the repeaters 'enter school at 7 or 8 years. 
The above conclusions are highly significant and seem to confirm the hypothesis of homogeneous^ 
behaviour by compensation, as.in a case like Colombia whcire the aggregate rates lead to a predic 
tion of survival by grades with a rather negligible margin of error. 

The concepts debated by Professor Brimer in his paper ("The Quantification of School 
Events Related to Educational Wastage") are based upon the optimum disaggregation of data for 
relevant decision ^making. The paper presents a model where wastage factors are classified on 
specific bases so as to assess efficiency at .ts various points and directed towards achievement 
evalu^ation. * \^ 

It is to be noted that most of thii parameters in the model could be estimated only if: 

(a) Some type of cohort coding method is introduced and, 

(b) two data collection points are fixed yearly. 

The model could contribute to the m6re refined evaluation of wastage and the relative importance 
of its factors and would be particularly useful in the case of^^chool systems without repetitions 
•(i. e. with automatic promotion). ^ ' 

In addition, the paper dealt with the critical question of how one can distinguish between 
the profitable and unprofjilable investment of time pejf pupil. Another important feature is the 
consideration that the production of estimates of wastage and inefficiency must relate to the eda> 
cational action which oqght to follpw through the medium of policy decision. "To know that wastage 
is high at a particular level is'not enough, unless it also locates the specific sector of the system 
where it occurs in such a way that action can be taken". ^ 
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I 

Two major ^reas are covered in the conclusions from the Expert meeting 1/ as follows: 

1 The improvement of indicators and analyses to be derived from the 
type of data currently available. 

"2. The need to. obtain a more detailed and comprehensive efficiency mea- 
sure. ^ 

J As regards item 1, the following conclusions were established: 

(a) The basic plausibility of the assumptions being adopted for the con- 
struction of cohorts on the basis. of observed rates was admitted. 

(b) The considerably better insight into the internal functioning of educa- 
tional systems provided through llow analyses as opposed to "apparent 
methods" was recognized and the misleading conclusions led to by the 
latter emphasized. 

(c) The needjor s^zpplementary indices was expressed as well as for the . 
devising of t^'pcldgies of wastage profiles. The need for ii^vestigaiion 
of the causal aspects, particularly social backgrounds of unsuccessful 
pupils, teachers* qualifications and available facilities, was empha- 
sized. In this connection, the convenience of studies aiming at com- 
paring different zones, districts or regions inside a^ountry was 

J stressed, taking into account the existing constratffts and limitations. 
These studies should lead to a typology of countries according ^to their 
^ efficiency features. 

(d) The interest provided by the simulation work on costing wastage was 
underlined and the desirability of application to more refined situations 

crecommended. The cost of measures to improve .efficiency was deemed 
deserving of careful investigation. 

(e) The participants expressed their appreciation of methods of predicting 
Wastage 'both as verification of t^e correct reconstruction of school 
cohorts and as forecasting tools. Several improvements were, however, 
suggested, '^especially th^ ^^nsideration of the rate of inrreast? of the 

c V number of newcomers entering the system in relevant cases 

In connection with item 2, the improvement of detail and coverage for a correct evaluation of 
«*fficiency, ilie main points discussed were: 

(a) The need for a closer identification of wastage events was underlined. 
In this connection, the evaluation of effectiveness of the educational 
^ , system at the different' stages and levels was judged to be of paramount 

importance. The establishment of enrolment flows through^ the various 
cycles, including drop-outs within t'- ^ school year, appeared highly 
desir^able. 



(b) The consideration of certain parameters, , so far not available (such as 
mortality, transfers Avithin the system, temporary interruption and 
later resumption, etc, ) were recummended. 

/ 
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(c) The identification of transition patterns between the various cycles and, 
th.e efficiency features-m-i«rgher education were judged to call for special 
sampling studies in selected countries. 

(d) The analysis of school careers. 
C. Follow-up 

Three major steps are the main recommendations from the expert meeting: 

(a| Methodological studies should be undertaken with a view to reaching an 
I acceptable precision of diagnosis in those countries where the minimum 
J^equired data are not available at present. Thus margins of error bet- 
ween actual flowSi reconstructed cohorts and apparent methods should 
be estimated by using enrolment by grade only and also by using age 
data when available. 

(b) Case studies should be undertaken "specially oriented towards the testing 
of some simple cohort coding system. 

(c) The above studies should permit the checking of indices derived by cru- 

\ <^er methods and th^ estimation of their level of accuracy. Factors .such 

as availability of places and successful completion should be taken into 
account in these exercises, 

/ ' Concluding renriarks 

The importance of a.s^essing tbe internal efficiency of an educational system cannot be overempha- 
sized Beyond the interi^t irrknatysing the present and past working of the system and thus eva- 
luating performance and inefficient critical points, such an analysis is crucial if one wants to have 
a look into the future - for instance to predict the implications of given change in educational policy 
in terms of potential intake and output at various stages. . 

This is precisely the type of problem being faced at present by Unesco Office of Statistics 
within its Education Projections Programme. A major problem is the determination of specific 
coefficients of educational survival and transition to reconstruct the trends within the system. The 
data required (enrolment by grade, sex and age and repeaters by grade and sex) is made available 
by a relatively large number of countries but there is evidence that statistical data essential for a 
proper appraisal of the) efficiency pattern could be obtained in other countries and also for supple- 
mentary aspects of this\problem, provided that resources are allocated and personnel are trained 
for this purpose. 
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